[01:20:55] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [01:25:55] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [03:43:52] 10PAWS: [feature] Can not log into PAWS today due to timeout - https://phabricator.wikimedia.org/T405183 (10Jarekt) 03NEW [03:50:52] 10PAWS: Request login account - https://phabricator.wikimedia.org/T405184 (10Intobesa.bot) 03NEW [07:35:52] (03PS3) 10Filippo Giunchedi: wmcs_libs: add get_address_family [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189791 (https://phabricator.wikimedia.org/T404584) [07:35:52] (03PS4) 10Filippo Giunchedi: nfs: make add_server idempotent for service IP [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189792 (https://phabricator.wikimedia.org/T404584) [07:35:52] (03PS3) 10Filippo Giunchedi: wmcs_libs: add network_id to NeutronPort [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189866 (https://phabricator.wikimedia.org/T404584) [07:35:53] (03PS3) 10Filippo Giunchedi: wmcs_libs: add optional ttl to recordset_create [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189867 (https://phabricator.wikimedia.org/T404584) [07:35:54] (03PS3) 10Filippo Giunchedi: nfs: do DNS flip on network migration [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189868 [07:36:37] (03CR) 10Volans: [C:03+1] "LGTM, couple of nits inline" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189791 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [07:39:53] (03CR) 10CI reject: [V:04-1] nfs: do DNS flip on network migration [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189868 (owner: 10Filippo Giunchedi) [07:46:54] (03PS4) 10Filippo Giunchedi: wmcs_libs: add get_ip_address_family [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189791 (https://phabricator.wikimedia.org/T404584) [07:46:54] (03PS5) 10Filippo Giunchedi: nfs: make add_server idempotent for service IP [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189792 (https://phabricator.wikimedia.org/T404584) [07:46:54] (03PS4) 10Filippo Giunchedi: wmcs_libs: add network_id to NeutronPort [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189866 (https://phabricator.wikimedia.org/T404584) [07:46:55] (03PS4) 10Filippo Giunchedi: wmcs_libs: add optional ttl to recordset_create [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189867 (https://phabricator.wikimedia.org/T404584) [07:46:56] (03PS4) 10Filippo Giunchedi: nfs: do DNS flip on network migration [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189868 [07:47:25] (03CR) 10Filippo Giunchedi: "Thank you for the review! Added basic tests too" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189791 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [07:50:47] (03CR) 10CI reject: [V:04-1] nfs: do DNS flip on network migration [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189868 (owner: 10Filippo Giunchedi) [07:53:12] (03CR) 10Leila237: [C:03+1] made changes to the design of the actors page and the search suggestions box redesign: T404737 [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1189983 (owner: 10Leila237) [07:54:15] (03CR) 10Filippo Giunchedi: [C:03+2] wmcs_libs: add get_ip_address_family [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189791 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [07:58:22] (03CR) 10NkwadaNora: [C:03+1] made changes to the design of the actors page and the search suggestions box redesign: T404737 [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1189983 (owner: 10Leila237) [08:27:12] (03CR) 10Wandji collins: "Please update the commit message to include the ticket Id for this change." [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1189950 (owner: 10Ologuie Arlette) [08:38:35] 10PAWS: [bug] PAWS-Server does not start - https://phabricator.wikimedia.org/T405205 (10WIKI-ville13) 03NEW [08:48:11] 06cloud-services-team, 14Cloud-VPS (Debian Buster Deprecation): Buster VMs in cloud-vps PKI project - https://phabricator.wikimedia.org/T405017#11200572 (10elukey) >>! In T405017#11197702, @Andrew wrote: >>>! In T405017#11196353, @elukey wrote: >> Hey Andrew! These are used by deployment-prep's TLS certificat... [08:53:38] 06cloud-services-team, 10Data-Services, 06Community-Tech, 06Data-Engineering, and 2 others: Unexpected error "Subquery returns more than 1 row" on wiki replicas - https://phabricator.wikimedia.org/T404473#11200603 (10fnegri) [08:53:55] !log dcaro@acme paws START - Cookbook wmcs.vps.instance.force_reboot vm paws-127c-uwce57bvcgrt-node-3 (cluster eqiad1, project paws) [08:54:00] !log dcaro@acme paws END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127c-uwce57bvcgrt-node-3 (cluster eqiad1, project paws) [08:54:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [08:54:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [08:59:51] 10PAWS: [feature] Can not log into PAWS today due to timeout - https://phabricator.wikimedia.org/T405183#11200698 (10dcaro) 05Open→03Resolved p:05Triage→03High a:03dcaro I found paws node-3 stuck, but flagged as correct, I uncordoned and rebooted it, now pods start correctly in it. @Jarekt it shou... [09:02:39] (03CR) 10Eugene233: [C:03+2] made changes to the design of the actors page and the search suggestions box redesign: T404737 [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1189983 (owner: 10Leila237) [09:03:20] (03Merged) 10jenkins-bot: made changes to the design of the actors page and the search suggestions box redesign: T404737 [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1189983 (owner: 10Leila237) [09:04:49] 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki mswikiquote - https://phabricator.wikimedia.org/T404703#11200723 (10fnegri) [09:05:14] 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki tokwiki - https://phabricator.wikimedia.org/T404570#11200729 (10fnegri) [09:05:26] 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki tokwiki - https://phabricator.wikimedia.org/T404570#11200732 (10fnegri) p:05Triage→03Medium [09:05:34] 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki mswikiquote - https://phabricator.wikimedia.org/T404703#11200733 (10fnegri) p:05Triage→03Medium [09:07:45] 06cloud-services-team, 10Toolforge: Job not restarting despite liveness probe failures - https://phabricator.wikimedia.org/T400957#11200737 (10fnegri) a:05fnegri→03None [09:08:40] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [tools,nfs,infra] Address tools NFS getting stuck with processes in D state - https://phabricator.wikimedia.org/T404584#11200739 (10fnegri) [09:08:41] 10cloud-services-team (FY2025/26-Q1), 10Striker: Attaching Phabricator account to a second Developer account via Striker results in a fatal error - https://phabricator.wikimedia.org/T319500#11200741 (10fnegri) [09:08:47] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [components-api] Validate `SourceReference` components point to `SourceBuild` components - https://phabricator.wikimedia.org/T403287#11200743 (10fnegri) [09:09:30] 10PAWS: [bug] PAWS-Server does not start - https://phabricator.wikimedia.org/T405205#11200746 (10dcaro) I think this is the same as T405183 [09:09:41] david-caro opened https://github.com/toolforge/paws/pull/499 [09:09:47] 10PAWS: [bug] PAWS-Server does not start - https://phabricator.wikimedia.org/T405205#11200749 (10dcaro) →14Duplicate dup:03T405183 [09:09:49] 10PAWS: [feature] Can not log into PAWS today due to timeout - https://phabricator.wikimedia.org/T405183#11200751 (10dcaro) [09:10:38] 10cloud-services-team (FY2025/26-Q1): Remove KernelErrors alerts - https://phabricator.wikimedia.org/T404300#11200754 (10fnegri) 05Open→03Resolved a:03taavi [09:30:42] (03CR) 10Essa237: [C:03+1] made changes to the design of the actors page and the search suggestions box redesign: T404737 [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1189983 (owner: 10Leila237) [09:30:57] (03CR) 10NkwadaNora: [C:03+1] properly align the shared production text, that was out of position [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1187775 (owner: 10NkwadaNora) [09:52:17] (03PS1) 10Ologuie Arlette: upadted popular actor design card and actor label page .T404827 [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1190214 [10:04:13] (03PS1) 10Ologuie Arlette: upadted popular actor design card and actor label page .T404827 [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1190217 [10:08:46] 10Toolforge (Toolforge iteration 24): [tools,infra,k8s] scale up the cluster, specifically CPU - https://phabricator.wikimedia.org/T404726#11200893 (10dcaro) Thanks @akosiaris, this is very helpful :) So I propose then for limits/requests to do: * If the user specifies `--cpu/--memory`, use that as both reques... [10:12:37] (03PS1) 10Ologuie Arlette: upadted shared production page .T404824 [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1190221 [10:14:05] (03PS1) 10Ologuie Arlette: upadted shared production page .T404824 [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1190222 [10:14:31] (03open) 10dcaro: kubernetes.capacity: don't page yet [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/38 [10:15:39] (03approved) 10filippo: kubernetes.capacity: don't page yet [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/38 (owner: 10dcaro) [10:24:52] (03CR) 10David Caro: [C:03+1] "LGTM, feel free to ignore the nit" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189792 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [10:25:09] (03merge) 10dcaro: kubernetes.capacity: don't page yet [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/38 [11:04:53] (03open) 10nikerabbit: Remove tasks feed from #translatewiki [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/62 [11:17:51] (03PS1) 10NkwadaNora: resolving merge conflicts .1187775 [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1190248 [11:17:51] (03PS1) 10NkwadaNora: resolving merge conflicts, installed required eslint plugins,updated eslint.config.mjs file to ignore the .next directory and other build/generated files now when you run tox --fix or npx eslint . --fix, it should only lint the actual source code and not the generated build files .T404075 [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1190249 [11:23:46] (03CR) 10NkwadaNora: [C:03+1] resolving merge conflicts, installed required eslint plugins,updated eslint.config.mjs file to ignore the .next directory and other build/ge [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1190249 (owner: 10NkwadaNora) [11:30:15] !log godog@r5 testlabs START - Cookbook wmcs.nfs.add_server [11:30:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Testlabs/SAL [11:38:47] (03CR) 10David Caro: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189866 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [11:40:33] (03CR) 10David Caro: [C:03+1] "LGTM, just a question" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189867 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [11:41:18] !log godog@r5 testlabs END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) [11:41:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Testlabs/SAL [11:47:09] (03CR) 10David Caro: [C:03+1] wmcs_libs: add optional ttl to recordset_create (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189867 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [11:50:37] (03CR) 10David Caro: [C:03+1] wmcs_libs: add optional ttl to recordset_create (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189867 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [11:55:44] (03PS6) 10Filippo Giunchedi: nfs: make add_server idempotent for service IP [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189792 (https://phabricator.wikimedia.org/T404584) [11:55:44] (03PS5) 10Filippo Giunchedi: wmcs_libs: add network_id to NeutronPort [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189866 (https://phabricator.wikimedia.org/T404584) [11:55:44] (03PS5) 10Filippo Giunchedi: wmcs_libs: add optional ttl to recordset_create [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189867 (https://phabricator.wikimedia.org/T404584) [11:55:45] (03PS5) 10Filippo Giunchedi: nfs: do DNS flip on network migration [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189868 [11:55:48] (03CR) 10Filippo Giunchedi: nfs: make add_server idempotent for service IP (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189792 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [11:57:03] (03CR) 10David Caro: wmcs_libs: add optional ttl to recordset_create (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189867 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [11:57:16] (03CR) 10CI reject: [V:04-1] wmcs_libs: add optional ttl to recordset_create [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189867 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [11:57:49] (03CR) 10David Caro: wmcs_libs: add optional ttl to recordset_create (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189867 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [11:58:09] !log godog@r5 testlabs START - Cookbook wmcs.nfs.add_server [11:58:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Testlabs/SAL [11:59:35] (03CR) 10CI reject: [V:04-1] nfs: do DNS flip on network migration [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189868 (owner: 10Filippo Giunchedi) [12:02:24] (03CR) 10David Caro: nfs: do DNS flip on network migration (032 comments) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189868 (owner: 10Filippo Giunchedi) [12:04:35] (03CR) 10Leila237: [C:03+1] resolving merge conflicts, installed required eslint plugins,updated eslint.config.mjs file to ignore the .next directory and other build/ge [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1190249 (owner: 10NkwadaNora) [12:05:41] (03CR) 10David Caro: nfs: do DNS flip on network migration (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189868 (owner: 10Filippo Giunchedi) [12:06:42] (03CR) 10David Caro: "Added a suggestion for the recordset_delete, unfortunately, the openstack cli is a bit inconsistent on where it implements the json flag o" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189868 (owner: 10Filippo Giunchedi) [12:08:45] !log godog@r5 testlabs END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) [12:08:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Testlabs/SAL [12:13:10] (03update) 10dcaro: resources: reduce the default cpu k8s request [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/215 [12:13:20] (03PS2) 10Ologuie Arlette: arrange popular actors design and compare page design .T404827 [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1189950 [12:14:06] 10PAWS: [feature] Can not log into PAWS today due to timeout - https://phabricator.wikimedia.org/T405183#11201189 (10Jarekt) It worked this time. Thanks [12:14:36] (03CR) 10Filippo Giunchedi: [C:03+2] nfs: make add_server idempotent for service IP [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189792 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [12:16:59] david-caro closed https://github.com/toolforge/paws/pull/499 [12:17:24] (03CR) 10Filippo Giunchedi: wmcs_libs: add optional ttl to recordset_create (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189867 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [12:18:59] (03Abandoned) 10Filippo Giunchedi: wmcs_libs: add optional ttl to recordset_create [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189867 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [12:19:25] (03CR) 10Filippo Giunchedi: [C:03+2] wmcs_libs: add network_id to NeutronPort [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189866 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi) [12:27:17] (03update) 10dcaro: Expand reuse_from validation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/126 (https://phabricator.wikimedia.org/T403287) (owner: 10damian) [12:29:30] (03merge) 10dcaro: Expand reuse_from validation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/126 (https://phabricator.wikimedia.org/T403287) (owner: 10damian) [12:32:14] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.160-20250922122936-8b99b504 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/975 (https://phabricator.wikimedia.org/T403287) [12:33:00] (03PS1) 10Ologuie Arlette: solve merge conflict .T404827 [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1190270 [12:34:02] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [12:36:47] 06cloud-services-team, 10Toolforge: toolforge logs appears to suffer from intermittent latency - https://phabricator.wikimedia.org/T402736#11201242 (10dcaro) I think this will be fixed with {T404176} [12:39:09] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [12:40:19] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [12:45:32] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [12:51:33] (03approved) 10dcaro: components-api: bump to 0.0.160-20250922122936-8b99b504 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/975 (https://phabricator.wikimedia.org/T403287) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:51:37] (03merge) 10dcaro: components-api: bump to 0.0.160-20250922122936-8b99b504 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/975 (https://phabricator.wikimedia.org/T403287) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:56:30] (03PS1) 10Ologuie Arlette: update shares actor from movie page .T404824 [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1190276 [12:59:53] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [components-api] Validate `SourceReference` components point to `SourceBuild` components - https://phabricator.wikimedia.org/T403287#11201322 (10dcaro) 05In progress→03Resolved [12:59:56] 10Toolforge (Toolforge iteration 24): [tools,infra,k8s] scale up the cluster, specifically CPU - https://phabricator.wikimedia.org/T404726#11201323 (10dcaro) a:03dcaro [13:01:23] (03update) 10dcaro: loki.alloy: decrease frequency for fetching logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/962 [13:10:25] david-caro closed https://github.com/toolforge/paws/pull/487 [13:34:26] (03CR) 10Filippo Giunchedi: nfs: do DNS flip on network migration (032 comments) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189868 (owner: 10Filippo Giunchedi) [13:44:23] (03CR) 10David Caro: nfs: do DNS flip on network migration (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1189868 (owner: 10Filippo Giunchedi) [13:51:59] 10PAWS: sql command line tools is broken - https://phabricator.wikimedia.org/T405229 (10Bugreporter) 03NEW [13:56:59] (03PS7) 10David Caro: vps.instance.force_reboot: add cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186443 [13:59:22] 10Toolforge (Toolforge iteration 24): [tools,infra,k8s] scale up the cluster, specifically CPU - https://phabricator.wikimedia.org/T404726#11201575 (10fgiunchedi) Thank you for the great summary, I can't meaningfully comment re: k8s specifics, however: >>! In T404726#11200892, @dcaro wrote: > Summarizing in a p... [14:01:05] 10PAWS: sql command line tools is broken - https://phabricator.wikimedia.org/T405229#11201582 (10Lucas_Werkmeister_WMDE) Apparently the branch name inside `install-sql-tool` needs to be changed from `master` to `main`. [14:13:10] (03PS6) 10David Caro: global: minor cleanups [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186483 [14:19:12] (03PS7) 10David Caro: global: minor cleanups [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186483 [14:19:24] (03CR) 10David Caro: global: minor cleanups (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186483 (owner: 10David Caro) [14:25:56] (03CR) 10David Caro: global: minor cleanups (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186483 (owner: 10David Caro) [14:41:22] 06cloud-services-team, 10Toolforge, 06SRE-OnFire, 10Sustainability (Incident Followup): [k8s,infra,o11y] Add paging alert when many tools are unreachable - https://phabricator.wikimedia.org/T399870#11201754 (10taavi) a:03taavi [14:51:24] (03open) 10taavi: kubernetes: Add alert on web service 5xx rate [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/39 (https://phabricator.wikimedia.org/T399870) [14:51:27] (03update) 10taavi: kubernetes: Add alert on web service 5xx rate [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/39 (https://phabricator.wikimedia.org/T399870) [14:51:47] (03update) 10taavi: kubernetes: Add alert on web service 5xx rate [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/39 (https://phabricator.wikimedia.org/T399870) [15:03:23] (03close) 10damian: _resolve_ref - use GitPython [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/128 [15:03:55] (03update) 10damian: Add validated type for git urls [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/121 [15:12:50] (03CR) 10Majavah: [C:04-1] "Please test your changes locally first before asking others to take the time to review your work." [labs/striker] - 10https://gerrit.wikimedia.org/r/1189915 (https://phabricator.wikimedia.org/T359554) (owner: 10Arendpieter) [15:29:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [15:32:05] (03update) 10taavi: kubernetes: Add alert on web service 5xx rate [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/39 (https://phabricator.wikimedia.org/T399870) [15:34:02] (03update) 10taavi: kubernetes: Add alert on web service 5xx rate [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/39 (https://phabricator.wikimedia.org/T399870) [15:34:15] (03update) 10dcaro: Add validated type for git urls [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/121 (owner: 10damian) [15:34:30] (03update) 10taavi: kubernetes: Add alert on web service 5xx rate [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/39 (https://phabricator.wikimedia.org/T399870) [15:34:32] (03merge) 10taavi: kubernetes: Add alert on web service 5xx rate [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/39 (https://phabricator.wikimedia.org/T399870) [15:36:14] (03approved) 10fnegri: resources: reduce the default cpu k8s request [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/215 (owner: 10dcaro) [15:48:59] (03update) 10dcaro: Add validated type for git urls [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/121 (owner: 10damian) [15:59:17] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [15:59:21] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [15:59:47] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [15:59:47] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [16:01:47] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [16:02:21] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [16:03:32] 06cloud-services-team, 10Toolforge: [envvars] ease revealing a secret - https://phabricator.wikimedia.org/T405024#11202292 (10dcaro) Quick note (will look the task better in a bit), you should avoid using the secrets in a command line, as that will be exposed to other users in the bastion. For mysql from the... [16:05:41] (03approved) 10dcaro: Add validated type for git urls [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/121 (owner: 10damian) [16:05:54] (03merge) 10dcaro: Add validated type for git urls [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/121 (owner: 10damian) [16:08:30] (03merge) 10dcaro: resources: reduce the default cpu k8s request [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/215 [16:09:32] 10Toolforge (Toolforge iteration 24): [tools,infra,k8s] scale up the cluster, specifically CPU - https://phabricator.wikimedia.org/T404726#11202365 (10akosiaris) >>! In T404726#11200892, @dcaro wrote: > Thanks @akosiaris, this is very helpful :) > > So I propose then for limits/requests to do: > > * If the use... [16:09:55] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.162-20250922160608-87287d65 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/976 [16:11:36] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [16:12:41] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.415-20250922160844-ff12d380 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/977 (https://phabricator.wikimedia.org/T404726) [16:16:31] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [16:16:41] 06cloud-services-team, 10Toolforge: [envvars] ease revealing a secret - https://phabricator.wikimedia.org/T405024#11202383 (10DamianZaremba) >>! In T405024#11202292, @dcaro wrote: > Quick note (will look the task better in a bit), you should avoid using the secrets in a command line, as that will be exposed to... [16:29:52] 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [tools,infra,k8s] scale up the cluster, specifically CPU - https://phabricator.wikimedia.org/T404726#11202421 (10dcaro) >> An issue here is that a lot of our workloads are cronjobs, so cronjobs being unable to trigger means most of the users don't get t... [16:32:50] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [16:32:53] 06cloud-services-team, 10Toolforge: [envvars] ease revealing a secret - https://phabricator.wikimedia.org/T405024#11202430 (10dcaro) >>! In T405024#11202383, @DamianZaremba wrote: >>>! In T405024#11202292, @dcaro wrote: >> Quick note (will look the task better in a bit), you should avoid using the secrets in a... [16:37:35] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [16:39:47] (03CR) 10David Caro: [C:03+2] vps.instance.force_reboot: add cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186443 (owner: 10David Caro) [16:43:45] (03Merged) 10jenkins-bot: vps.instance.force_reboot: add cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186443 (owner: 10David Caro) [16:44:34] (03approved) 10dcaro: components-api: bump to 0.0.162-20250922160608-87287d65 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/976 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:44:41] (03merge) 10dcaro: components-api: bump to 0.0.162-20250922160608-87287d65 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/976 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:47:21] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [16:47:21] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [16:49:48] 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [tools,infra,k8s] scale up the cluster, specifically CPU - https://phabricator.wikimedia.org/T404726#11202476 (10JJMC89) > This is a subjective matter, if your cronjob does not start for the next 5h, is it just a delay or considered so bad that it's jus... [16:50:29] 06cloud-services-team, 10Cloud-VPS, 10Ceph, 06DC-Ops, and 2 others: cloudcephosd1025 won't reimage - https://phabricator.wikimedia.org/T405258 (10Andrew) 03NEW [16:50:42] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [16:51:16] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [16:53:22] 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [tools,infra,k8s] scale up the cluster, specifically CPU - https://phabricator.wikimedia.org/T404726#11202498 (10dcaro) Hi @JJMC89, thanks for your comments. > If you are going to change the CPU/memory defaults or what the CPU and memory values transla... [17:03:43] 06cloud-services-team, 10Cloud-VPS, 10Ceph, 06DC-Ops, and 2 others: cloudcephosd1025 won't reimage - https://phabricator.wikimedia.org/T405258#11202559 (10Jclark-ctr) I see the Drives are not setup correctly and some list as foreign state. [17:10:30] (03PS2) 10Arendpieter: Use IDP for authentication [labs/striker] - 10https://gerrit.wikimedia.org/r/1189915 (https://phabricator.wikimedia.org/T359554) [17:11:23] (03CR) 10CI reject: [V:04-1] Use IDP for authentication [labs/striker] - 10https://gerrit.wikimedia.org/r/1189915 (https://phabricator.wikimedia.org/T359554) (owner: 10Arendpieter) [17:13:11] 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [tools,infra,k8s] scale up the cluster, specifically CPU - https://phabricator.wikimedia.org/T404726#11202598 (10JJMC89) >>! In T404726#11202498, @dcaro wrote: > Hi @JJMC89, thanks for your comments. > >> If you are going to change the CPU/memory defau... [17:13:26] (03CR) 10Arendpieter: Use IDP for authentication (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/1189915 (https://phabricator.wikimedia.org/T359554) (owner: 10Arendpieter) [17:20:59] (03open) 10damian: Expand `source` support in `ToolConfig` [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T402764) [17:26:28] 06cloud-services-team, 10Cloud-VPS, 10Ceph, 06DC-Ops, and 2 others: cloudcephosd1025 won't reimage - https://phabricator.wikimedia.org/T405258#11202643 (10Jclark-ctr) Cleared foreign configs and converted drives to non-raid [17:36:51] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [17:37:28] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [17:37:32] (03PS3) 10Arendpieter: Use IDP for authentication [labs/striker] - 10https://gerrit.wikimedia.org/r/1189915 (https://phabricator.wikimedia.org/T359554) [17:37:56] (03CR) 10CI reject: [V:04-1] Use IDP for authentication [labs/striker] - 10https://gerrit.wikimedia.org/r/1189915 (https://phabricator.wikimedia.org/T359554) (owner: 10Arendpieter) [17:38:40] (03update) 10damian: Expand `source` support in `ToolConfig` [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T402764) [17:42:08] (03close) 10damian: Add `source` support to ToolConfig [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/127 (https://phabricator.wikimedia.org/T402764) [17:46:36] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [components-api] allow specifying `source_repo`+`ref` for the config - https://phabricator.wikimedia.org/T402764#11202717 (10DamianZaremba) To close this out only https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api... [17:52:50] 06cloud-services-team, 10Toolforge: Build standard images under pack / support execution via components-api - https://phabricator.wikimedia.org/T405262 (10DamianZaremba) 03NEW [17:57:04] (03PS4) 10Arendpieter: Use IDP for authentication [labs/striker] - 10https://gerrit.wikimedia.org/r/1189915 (https://phabricator.wikimedia.org/T359554) [17:58:13] (03update) 10damian: Expand `source` support in `ToolConfig` [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T402764) [17:58:50] (03CR) 10CI reject: [V:04-1] Use IDP for authentication [labs/striker] - 10https://gerrit.wikimedia.org/r/1189915 (https://phabricator.wikimedia.org/T359554) (owner: 10Arendpieter) [17:58:58] 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [tools,infra,k8s] scale up the cluster, specifically CPU - https://phabricator.wikimedia.org/T404726#11202736 (10akosiaris) >>! In T404726#11202421, @dcaro wrote: >>> An issue here is that a lot of our workloads are cronjobs, so cronjobs being unable to... [18:01:52] (03update) 10damian: Expand `source` support in `ToolConfig` [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T402764) [18:10:19] (03PS5) 10Arendpieter: Use IDP for authentication [labs/striker] - 10https://gerrit.wikimedia.org/r/1189915 (https://phabricator.wikimedia.org/T359554) [18:12:31] 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [tools,infra,k8s] scale up the cluster, specifically CPU - https://phabricator.wikimedia.org/T404726#11202759 (10akosiaris) >>! In T404726#11202598, @JJMC89 wrote: >>>! In T404726#11202498, @dcaro wrote: >> Hi @JJMC89, thanks for your comments. >> >>>... [18:15:28] (03CR) 10Arendpieter: "All tests are now passing." [labs/striker] - 10https://gerrit.wikimedia.org/r/1189915 (https://phabricator.wikimedia.org/T359554) (owner: 10Arendpieter) [18:16:21] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [18:19:53] PROBLEM - Host cloudcephosd1024 is DOWN: PING CRITICAL - Packet loss = 100% [18:21:21] RECOVERY - Host cloudcephosd1024 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [18:22:31] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [19:09:34] 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [tools,infra,k8s] scale up the cluster, specifically CPU - https://phabricator.wikimedia.org/T404726#11202888 (10JJMC89) >>! In T404726#11202759, @akosiaris wrote: >>>! In T404726#11202598, @JJMC89 wrote: >>>>! In T404726#11202498, @dcaro wrote: >>> Hi... [19:15:57] (03update) 10raymond-ndibe: [tool-config] handle unset and default arguments consistently [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/123 (https://phabricator.wikimedia.org/T401648 https://phabricator.wikimedia.org/T402572) [19:18:29] (03update) 10raymond-ndibe: [tool-config] handle unset and default arguments consistently [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/123 (https://phabricator.wikimedia.org/T401648 https://phabricator.wikimedia.org/T402572) [19:31:02] (03update) 10raymond-ndibe: [tool-config] handle unset and default arguments consistently [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/123 (https://phabricator.wikimedia.org/T401648 https://phabricator.wikimedia.org/T402572) [19:40:12] (03update) 10raymond-ndibe: [config] support port protocol [repos/cloud/toolforge/components-api] (handle_unset_and_default_arguments_consistently) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/119 (https://phabricator.wikimedia.org/T401994) [19:51:00] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [19:53:36] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [20:01:46] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [20:02:28] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [20:06:38] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [20:09:38] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [20:10:15] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [20:14:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [20:17:31] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [20:17:43] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [20:18:23] PROBLEM - Host cloudcephosd1026 is DOWN: PING CRITICAL - Packet loss = 100% [20:19:09] RECOVERY - Host cloudcephosd1026 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [20:20:16] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [20:24:18] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [20:24:38] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [20:33:20] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [20:34:05] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [20:41:12] !log component-configs tools.cluebotng-review Test migrating log to feed channel [20:41:13] wm-bot2: Unknown project "component-configs" [20:43:26] !log damian-scripts@tools-bastion-15 tools.cluebotng-review Test migrating log to feed channel [20:43:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [20:43:46] !log component-configs@tools-bastion tools.cluebotng-review Test migrating log to feed channel [20:43:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [21:05:16] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213