[01:09:44] 10Cloud-VPS (Project-requests): Request creation of futureaudiences VPS project - https://phabricator.wikimedia.org/T389158#10650049 (10derenrich) Yes I had read that before making this application but it sounded those were not necessarily hard rules. Given FA's velocity and diverse set of projects it seems un... [02:12:07] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudweb.set_maintenance (T381499) [02:12:14] T381499: Upgrade cloud-vps openstack to version 'Dalmation' - https://phabricator.wikimedia.org/T381499 [02:12:49] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) (T381499) [02:15:43] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' [02:18:06] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet' [02:18:28] 06cloud-services-team, 10Toolforge: Analyze Toolforge and Toolsbeta for Virtual Resource Usage - https://phabricator.wikimedia.org/T389081#10650109 (10Chuckonwumelu) [02:28:07] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Upgrade cloud-vps openstack to version 'Dalmatian' - https://phabricator.wikimedia.org/T381499#10650127 (10Andrew) [02:54:00] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' [03:06:09] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet' [04:08:56] 06cloud-services-team, 10Cloud-VPS: Upgrade cloud-vps openstack to version 'Dalmatian' - https://phabricator.wikimedia.org/T381499#10650276 (10Andrew) Trove's db migration table seems broken, I have a somewhat repaired one in cloudcontrol2004-dev:~andrew/5c68b4fb3cd1_add_datastore_version_registry_extension.py [04:08:58] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' [04:09:17] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet' [04:09:38] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' [04:09:56] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet' [04:10:48] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' [04:11:08] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet' [04:11:47] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' [04:13:31] (03open) 10raymond-ndibe: Draft: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] (refactor_imagename_field_in_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118) [04:29:04] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' [04:29:11] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' [04:38:30] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet' [04:39:38] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet' [04:40:30] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' [04:49:49] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet' [04:53:20] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' [04:54:00] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet' [04:56:32] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet' [04:58:29] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' [04:58:48] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet' [05:00:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet' [05:04:19] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' [05:04:37] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet' [05:06:04] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' [05:06:07] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet' [05:07:33] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet' [05:08:59] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' [05:09:03] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet' [05:12:36] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' [05:12:45] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) on host 'cloudnet2005-dev.codfw.wmnet' [05:13:08] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' [05:13:35] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet' [05:14:06] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' [05:14:26] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet' [05:14:52] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' [05:15:02] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet' [05:15:53] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' [05:16:02] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet' [05:17:04] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' [05:17:24] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet' [05:17:37] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' [05:17:59] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet' [05:18:39] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' [05:18:59] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet' [05:20:13] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' [05:20:34] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet' [05:20:46] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' [05:20:53] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet' [05:21:51] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' [05:22:31] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet' [05:23:11] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' [05:23:34] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet' [05:25:14] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' [05:25:25] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet' [05:25:51] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' [05:26:11] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet' [05:27:28] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' [05:27:47] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet' [05:28:42] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [05:30:20] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' [05:30:41] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet' [05:31:41] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' [05:31:59] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [05:32:01] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet' [05:33:57] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' [05:34:00] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet' [05:34:06] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' [05:34:25] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2005-dev.codfw.wmnet' [05:36:20] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' [05:37:29] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' [05:46:01] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet' [05:46:24] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' [05:46:38] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet' [05:46:47] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet' [05:47:06] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet' [05:47:35] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' [05:47:57] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet' [05:48:52] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' [05:48:54] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet' [05:49:05] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' [05:49:29] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudnet2006-dev.codfw.wmnet' [05:50:12] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' [05:54:13] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet' [05:54:24] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet' [05:59:58] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' [06:00:33] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet' [06:01:47] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet' [06:01:55] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet' [06:05:15] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet' [06:05:23] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet' [06:08:21] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet' [06:11:06] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [06:12:04] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet' [06:14:01] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [06:21:12] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [06:24:07] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [08:00:13] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [10:28:55] 06cloud-services-team, 10Toolforge: [components-api] Add "runs" section to the deployment structure - https://phabricator.wikimedia.org/T389339 (10dcaro) 03NEW [10:49:39] (03PS1) 10Lokas: Bug: T225816 Add support for right to left (RTL) languages to ISA [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1129211 (https://phabricator.wikimedia.org/T225816) [10:49:39] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [10:50:37] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [10:51:14] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [10:51:45] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [10:55:41] (03PS2) 10Lokas: Add support for right to left (RTL) languages to ISA [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1129211 (https://phabricator.wikimedia.org/T225816) [10:56:23] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [10:58:35] (03CR) 10GergesShamon: "recheck" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1129211 (https://phabricator.wikimedia.org/T225816) (owner: 10Lokas) [10:58:53] (03CR) 10CI reject: [V:04-1] Add support for right to left (RTL) languages to ISA [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1129211 (https://phabricator.wikimedia.org/T225816) (owner: 10Lokas) [11:00:31] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [11:08:16] (03PS3) 10Amire80: Add support for right to left (RTL) languages to ISA [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1129211 (https://phabricator.wikimedia.org/T225816) (owner: 10Lokas) [11:08:35] (03CR) 10CI reject: [V:04-1] Add support for right to left (RTL) languages to ISA [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1129211 (https://phabricator.wikimedia.org/T225816) (owner: 10Lokas) [11:11:59] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [11:12:35] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [11:16:25] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [11:17:13] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [11:20:20] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [11:22:23] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [11:27:10] (03update) 10aborrero: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [11:28:58] 06cloud-services-team, 10Toolforge: [components-api] Add "runs" section to the deployment structure - https://phabricator.wikimedia.org/T389339#10651332 (10dcaro) p:05Triage→03Medium [11:34:04] (03update) 10aborrero: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [11:58:09] (03PS4) 10Lokas: Add support for right to left (RTL) languages to ISA [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1129211 (https://phabricator.wikimedia.org/T225816) [11:58:50] (03PS5) 10Lokas: Add support for right to left (RTL) languages to ISA [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1129211 (https://phabricator.wikimedia.org/T225816) [12:00:24] (03CR) 10GergesShamon: "recheck" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1129211 (https://phabricator.wikimedia.org/T225816) (owner: 10Lokas) [12:01:09] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,task_id,no_dologmsg,cluster_name,all_services,nova,glance,keystone,cinder,neutron,trove,magnum,heat,swift,designate,filter_nodes [12:01:11] (03CR) 10GergesShamon: [C:03+1] "thanks" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1129211 (https://phabricator.wikimedia.org/T225816) (owner: 10Lokas) [12:02:12] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,task_id,no_dologmsg,cluster_name,all_services,nova,glance,keystone,cinder,neutron,trove,magnum,heat,swift,designate,filter_nodes [12:10:06] 10Tool-bub2, 07Documentation: Documentation for BUB2 Tool - https://phabricator.wikimedia.org/T364082#10651579 (10Aklapper) [12:10:09] 10Tool-bub2, 13Patch-For-Review, 07Test-Coverage: Write unit test cases - https://phabricator.wikimedia.org/T344117#10651581 (10Aklapper) [12:20:25] (03PS1) 10Arturo Borrero Gonzalez: wmcs.openstack.restart_openstack: fix description if only a few services were set [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129233 [12:23:50] (03CR) 10CI reject: [V:04-1] wmcs.openstack.restart_openstack: fix description if only a few services were set [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129233 (owner: 10Arturo Borrero Gonzalez) [12:24:29] (03PS2) 10Arturo Borrero Gonzalez: wmcs.openstack.restart_openstack: fix description if only a few services were set [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129233 [12:25:29] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10651624 (10aborrero) today the eqiad1 deploymetn was a NOOP with the refactored code. [12:27:52] (03CR) 10CI reject: [V:04-1] wmcs.openstack.restart_openstack: fix description if only a few services were set [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129233 (owner: 10Arturo Borrero Gonzalez) [12:32:29] (03PS3) 10Arturo Borrero Gonzalez: wmcs.openstack.restart_openstack: fix description if only a few services were set [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129233 [12:52:20] 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [jobs-api] refactor models - https://phabricator.wikimedia.org/T389118#10651726 (10dcaro) 05Open→03In progress [12:53:24] (03update) 10dcaro: deployment: time out stuck deployment after 1h [repos/cloud/toolforge/components-api] (shorten_deploy_id) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/59 [13:08:39] (03update) 10dcaro: deployment: time out stuck deployment after 1h [repos/cloud/toolforge/components-api] (shorten_deploy_id) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/59 [13:17:20] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10651825 (10aborrero) 05In progress→03Stalled blocked on codfw1dev designate being unavailable and code review for https://gitlab.wikimedia.org/repos... [13:21:17] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: tofu-infra: add support for DNS zones created by wmfkeystonehook - https://phabricator.wikimedia.org/T376110#10651848 (10aborrero) [13:21:55] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: tofu-infra: add support for DNS zones created by wmfkeystonehook - https://phabricator.wikimedia.org/T376110#10651851 (10aborrero) depending on the results of parent task {T385604} we may want to decline this ticket. [13:27:04] (03CR) 10Arturo Borrero Gonzalez: [C:03+2] wmcs.openstack.restart_openstack: fix description if only a few services were set [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129233 (owner: 10Arturo Borrero Gonzalez) [13:28:50] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [components-api] Add "runs" section to the deployment structure - https://phabricator.wikimedia.org/T389339#10651875 (10dcaro) [13:29:07] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [components-api] Add "runs" section to the deployment structure - https://phabricator.wikimedia.org/T389339#10651877 (10dcaro) a:03dcaro [14:25:22] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate [14:26:25] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for service: project,designate [14:52:22] (03PS1) 10Arturo Borrero Gonzalez: wmcs.openstack.restart_openstack: ignore project argument when reporting operations [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129291 [14:54:06] 06cloud-services-team, 10Toolforge: Analyze Toolforge and Toolsbeta for Virtual Resource Usage - https://phabricator.wikimedia.org/T389081#10652420 (10aborrero) [14:55:56] 06cloud-services-team: Onboard Chuck Onwumelu - https://phabricator.wikimedia.org/T386715#10652439 (10aborrero) hey @Chuckonwumelu this item ` Add to cloud-wide root (put up a patchset to modules/passwords/templates/root-authorized-keys.erb in https://gerrit.wikimedia.org/r/#/admin/projects/labs/private)` I thin... [14:56:30] (03CR) 10CI reject: [V:04-1] wmcs.openstack.restart_openstack: ignore project argument when reporting operations [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129291 (owner: 10Arturo Borrero Gonzalez) [14:57:07] (03PS2) 10Arturo Borrero Gonzalez: wmcs.openstack.restart_openstack: ignore project argument when reporting operations [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129291 [15:45:06] (03PS1) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) [15:50:17] (03CR) 10CI reject: [V:04-1] wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez) [15:56:09] (03PS2) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) [15:58:47] (03PS3) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) [16:04:17] (03CR) 10CI reject: [V:04-1] wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez) [16:13:49] (03PS4) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) [17:36:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-19 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [17:58:47] 06cloud-services-team, 10Data-Services, 10Projects-Cleanup: Archive the operations/debs/bdsync repository - https://phabricator.wikimedia.org/T377882#10653589 (10Aklapper) [18:16:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-19 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [18:20:09] 10VPS-project-Codesearch, 06collaboration-services, 13Patch-For-Review: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199#10653808 (10Dzahn) 05Open→03In progress [19:22:40] (03PS2) 10Umherirrender: Add function documentation [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1119843 [20:03:01] (03update) 10raymond-ndibe: Draft: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] (refactor_imagename_field_in_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118) [20:13:30] (03update) 10dcaro: deployment: time out stuck deployment after 1h [repos/cloud/toolforge/components-api] (shorten_deploy_id) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/59 [20:18:38] (03update) 10raymond-ndibe: Draft: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] (refactor_imagename_field_in_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118) [20:18:53] (03update) 10raymond-ndibe: Draft: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] (refactor_imagename_field_in_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118) [20:19:04] (03update) 10raymond-ndibe: Draft: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] (refactor_imagename_field_in_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118) [20:37:16] (03update) 10dcaro: deployment: time out stuck deployment after 1h [repos/cloud/toolforge/components-api] (shorten_deploy_id) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/59 [20:38:32] (03update) 10dcaro: deployment: time out stuck deployment after 1h [repos/cloud/toolforge/components-api] (shorten_deploy_id) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/59 [21:02:38] (03PS1) 10Dzahn: vrts: add profile::vrts::exim_deny_senders with fake value [labs/private] - 10https://gerrit.wikimedia.org/r/1129374 [21:03:38] (03PS2) 10Dzahn: vrts: add profile::vrts::exim_deny_senders with fake value [labs/private] - 10https://gerrit.wikimedia.org/r/1129374 (https://phabricator.wikimedia.org/T389079) [21:03:51] (03CR) 10Dzahn: [V:03+2 C:03+2] vrts: add profile::vrts::exim_deny_senders with fake value [labs/private] - 10https://gerrit.wikimedia.org/r/1129374 (https://phabricator.wikimedia.org/T389079) (owner: 10Dzahn) [21:29:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-19 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [22:06:39] 10wikitech.wikimedia.org, 06MediaWiki-Platform-Team, 05SUL3: Fix stuck old cookies on Wikitech - https://phabricator.wikimedia.org/T389433 (10Tgr) 03NEW [22:14:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-19 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [22:14:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-19 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [22:19:18] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-19 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [23:45:08] (03update) 10raymond-ndibe: Draft: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] (refactor_imagename_field_in_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118) [23:51:18] (03update) 10raymond-ndibe: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] (refactor_imagename_field_in_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118)