[00:21:54] 06cloud-services-team, 10wikitech.wikimedia.org, 10Wikimedia-Extension-setup, 07Documentation, 07I18n: Install Translate extension on wikitech - https://phabricator.wikimedia.org/T100313#10194419 (10Bugreporter) >>! In T100313#10194214, @bd808 wrote: >>>! In T100313#10191749, @Bugreporter wrote: >> This... [00:46:24] 10wikitech.wikimedia.org: MABot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376222#10194454 (10Pppery) FYI MarcoAurelio has not been very active lately so it may be a while before they see this. [01:20:17] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10194503 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm [02:16:04] 06cloud-services-team, 10Cloud-VPS, 06collaboration-services: puppet problems mounting cinder volumes (and suggested fixes) - https://phabricator.wikimedia.org/T371573#10194537 (10Dzahn) 05Open→03Resolved I tested this on a fresh instance when applying the puppet role for the first time. The mount w... [02:40:37] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10194558 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm executed... [02:50:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:05:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:35:48] 10wikitech.wikimedia.org: Toolforge Image Bot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376223#10194659 (10taavi) a:03taavi [07:58:04] 10Cloud-VPS (Project-requests): Request creation of catalyst-dev VPS project - https://phabricator.wikimedia.org/T376211#10194824 (10dcaro) a:03fnegri +1 approved [07:59:10] (03approved) 10dcaro: Increase memory quota for "sqid" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/543 (https://phabricator.wikimedia.org/T375070) (owner: 10fnegri) [08:07:54] (03approved) 10aborrero: Increase memory quota for "sqid" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/543 (https://phabricator.wikimedia.org/T375070) (owner: 10fnegri) [08:30:55] 10Cloud-Services, 10wikitech.wikimedia.org, 10observability: Monitor for wikitech logins failing - https://phabricator.wikimedia.org/T91226#10194853 (10taavi) 05Open→03Declined Wikitech is less of a special case these days so I don't think that is needed [08:33:12] 10Striker, 10wikitech.wikimedia.org, 10MediaWiki-extensions-OATHAuth: Wikitech 2FA does not appear to allow recovery with recovery codes - https://phabricator.wikimedia.org/T204682#10194857 (10taavi) 05Open→03Invalid No. [08:37:23] 10wikitech.wikimedia.org, 06DBA, 07Wikimedia-database-issue, 07Wikimedia-production-error: Wikimedia\Rdbms\DBUnexpectedError: Database servers in cluster30 are overloaded. In order to protect application servers, the circuit breaking to databases of this section ha... - https://phabricator.wikimedia.org/T376249 [08:45:21] 10wikitech.wikimedia.org: Setting new password at wikitech does not work - https://phabricator.wikimedia.org/T376140#10194895 (10aborrero) 05Resolved→03Open I'm reopening because I believe I still see this problem happening for my account `Arturo Borrero Gonzalez`. But maybe the actual problem at this point... [08:52:31] (03update) 10fnegri: Increase memory quota for "sqid" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/543 (https://phabricator.wikimedia.org/T375070) [08:52:51] (03update) 10fnegri: Increase memory quota for "sqid" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/543 (https://phabricator.wikimedia.org/T375070) [08:53:57] (03merge) 10fnegri: Increase memory quota for "sqid" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/543 (https://phabricator.wikimedia.org/T375070) [08:59:17] 10Cloud-VPS (Project-requests): Request creation of catalyst-dev VPS project - https://phabricator.wikimedia.org/T376211#10194929 (10fnegri) 05Open→03In progress [09:02:04] (03open) 10aborrero: flavors: refactor into the same per-project layout as the rest of the repo [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/76 (https://phabricator.wikimedia.org/T375283) [09:02:54] 10wikitech.wikimedia.org, 06DBA, 07Wikimedia-database-issue, 07Wikimedia-production-error: Wikimedia\Rdbms\DBUnexpectedError: Database servers in cluster30 are overloaded. In order to protect application servers, the circuit breaking to databases of this s... - https://phabricator.wikimedia.org/T376249#10194942 [09:06:59] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [09:09:12] (03open) 10fnegri: Create project catalyst-dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 (https://phabricator.wikimedia.org/T376211) [09:10:47] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 (T376211) [09:10:53] T376211: Request creation of catalyst-dev VPS project - https://phabricator.wikimedia.org/T376211 [09:11:03] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 (T376211) [09:11:32] !log fnegri@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers [09:14:15] (03update) 10fnegri: Create project catalyst-dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 (https://phabricator.wikimedia.org/T376211) [09:14:15] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 (T376211) [09:14:45] !log fnegri@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 (T376211) [09:16:21] (03update) 10aborrero: flavors: refactor into the same per-project layout as the rest of the repo [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/76 (https://phabricator.wikimedia.org/T375283) [09:17:08] (03update) 10fnegri: Increase memory quota for "sqid" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/543 (https://phabricator.wikimedia.org/T375070) [09:18:31] 10wikitech.wikimedia.org, 06DBA, 07Wikimedia-database-issue, 07Wikimedia-production-error: Wikimedia\Rdbms\DBUnexpectedError: Database servers in cluster30 are overloaded. In order to protect application servers, the circuit breaking to databases of this s... - https://phabricator.wikimedia.org/T376249#10195004 [09:18:39] 10Toolforge: component.deploy cookbook fails for branch "main" - https://phabricator.wikimedia.org/T376254 (10fnegri) 03NEW [09:18:55] (03update) 10aborrero: flavors: refactor into the same per-project layout as the rest of the repo [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/76 (https://phabricator.wikimedia.org/T375283) [09:19:52] (03update) 10aborrero: flavors: refactor into the same per-project layout as the rest of the repo [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/76 (https://phabricator.wikimedia.org/T375283) [09:22:35] (03PS1) 10JMeybohm: kubernetes::worker_containerd: Fix registry_auth hiera key [labs/private] - 10https://gerrit.wikimedia.org/r/1077323 (https://phabricator.wikimedia.org/T362408) [09:25:08] 10wikitech.wikimedia.org, 06DBA, 07Wikimedia-database-issue, 07Wikimedia-production-error: Wikimedia\Rdbms\DBUnexpectedError: Database servers in cluster30 are overloaded. In order to protect application servers, the circuit breaking to databases of this s... - https://phabricator.wikimedia.org/T376249#10195024 [09:26:37] (03CR) 10JMeybohm: [V:03+2 C:03+2] kubernetes::worker_containerd: Fix registry_auth hiera key [labs/private] - 10https://gerrit.wikimedia.org/r/1077323 (https://phabricator.wikimedia.org/T362408) (owner: 10JMeybohm) [09:27:45] 10Tool-techcontribs: Do not show Phabricator milestones in the list of projects a user is a member of - https://phabricator.wikimedia.org/T375976#10195030 (10Chlod) p:05Triage→03Medium Will do in a bit after I finish some current pending work. Should be easily doable, since Phabricator provides that info in... [09:35:06] 10Toolforge (Quota-requests): Request increased quota for sqid Toolforge tool - https://phabricator.wikimedia.org/T375070#10195087 (10fnegri) 05Open→03Resolved [09:36:00] 10wikitech.wikimedia.org, 06DBA, 07Wikimedia-database-issue, 07Wikimedia-production-error: Wikimedia\Rdbms\DBUnexpectedError: Database servers in cluster30 are overloaded. In order to protect application servers, the circuit breaking to databases of this s... - https://phabricator.wikimedia.org/T376249#10195070 [09:36:35] (03open) 10dcaro: slavina/clean up code [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/11 [09:36:50] (03update) 10dcaro: slavina/clean up code [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/11 [09:38:50] 10Cloud-VPS, 07Documentation: Clean up Cloud VPS doc content and sequence for account / project / instance setup and access - https://phabricator.wikimedia.org/T347637#10195092 (10fnegri) [09:38:51] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 06Tech-Docs-Team, 07Documentation: WMCS: Document different types of root and admin privileges - https://phabricator.wikimedia.org/T375113#10195093 (10fnegri) [09:39:11] 10Cloud-VPS, 07Documentation: Clean up Cloud VPS doc content and sequence for account / project / instance setup and access - https://phabricator.wikimedia.org/T347637#10195095 (10fnegri) p:05Triage→03Medium [09:39:49] 10Cloud-VPS (Project-requests), 13Patch-For-Review: Request creation of catalyst-dev VPS project - https://phabricator.wikimedia.org/T376211#10195096 (10fnegri) p:05Triage→03High [09:43:38] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10195107 (10jcrespo) [09:44:34] 10Tool-techcontribs: Only show direct Phabricator project memberships - https://phabricator.wikimedia.org/T376257 (10Chlod) 03NEW [09:44:50] 10Tool-techcontribs: Only show direct Phabricator project memberships - https://phabricator.wikimedia.org/T376257#10195122 (10Chlod) p:05Triage→03Medium [09:46:41] 10wikitech.wikimedia.org, 06DBA, 07Wikimedia-database-issue, 07Wikimedia-production-error: Wikimedia\Rdbms\DBUnexpectedError: Database servers in cluster30 are overloaded. In order to protect application servers, the circuit breaking to databases of this s... - https://phabricator.wikimedia.org/T376249#10195106 [09:48:20] (03approved) 10sstefanova: slavina/clean up code [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/11 (owner: 10dcaro) [09:54:27] (03merge) 10sstefanova: slavina/clean up code [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/11 (owner: 10dcaro) [09:54:44] (03open) 10dcaro: add mypy [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/12 [09:56:14] (03approved) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/9 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:56:36] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 [09:57:19] (03merge) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/9 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:57:23] (03approved) 10aborrero: Create project catalyst-dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 (https://phabricator.wikimedia.org/T376211) (owner: 10fnegri) [09:58:55] 06cloud-services-team, 10Phabricator: Herald_rule_for_cloud_services_team - https://phabricator.wikimedia.org/T376263 (10fnegri) 03NEW [09:59:09] 10cloud-services-team (FY2024/2025-Q1-Q2): Test using phabricator-maintenance-bot to sync wmcs-related boards - https://phabricator.wikimedia.org/T358251#10195207 (10fnegri) [09:59:10] 06cloud-services-team, 10Phabricator: Herald_rule_for_cloud_services_team - https://phabricator.wikimedia.org/T376263#10195206 (10fnegri) [09:59:41] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 [10:01:25] (03merge) 10fnegri: Create project catalyst-dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 (https://phabricator.wikimedia.org/T376211) [10:03:14] (03update) 10dcaro: add mypy [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/12 [10:03:56] (03update) 10dcaro: global: add mypy checks and fixes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/12 [10:04:48] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch (T376211) [10:04:55] T376211: Request creation of catalyst-dev VPS project - https://phabricator.wikimedia.org/T376211 [10:05:24] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch (T376211) [10:10:11] (03PS1) 10David Caro: deploy: skip sending note to MR if no MR was found [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077336 [10:10:53] (03PS2) 10David Caro: deploy: skip sending note to MR if no MR was found [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077336 (https://phabricator.wikimedia.org/T376254) [10:12:18] (03CR) 10FNegri: [C:03+1] "Thanks for the fix!" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077336 (https://phabricator.wikimedia.org/T376254) (owner: 10David Caro) [10:15:16] (03CR) 10CI reject: [V:04-1] deploy: skip sending note to MR if no MR was found [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077336 (https://phabricator.wikimedia.org/T376254) (owner: 10David Caro) [10:15:53] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch (T376211) [10:16:00] T376211: Request creation of catalyst-dev VPS project - https://phabricator.wikimedia.org/T376211 [10:16:28] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch (T376211) [10:19:23] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error report - https://phabricator.wikimedia.org/T376267 (10jijiki) 03NEW [10:21:50] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch (T376211) [10:21:56] T376211: Request creation of catalyst-dev VPS project - https://phabricator.wikimedia.org/T376211 [10:25:11] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error report - https://phabricator.wikimedia.org/T376267#10195305 (10jijiki) [10:26:25] !log fnegri@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch (T376211) [10:28:58] FIRING: SystemdUnitDown: The service unit rsync_enterprise_htmldumps.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [10:29:08] 06cloud-services-team, 10Phabricator: Create Herald rule for cloud-services-team - https://phabricator.wikimedia.org/T376263#10195315 (10Aklapper) [10:31:47] (03open) 10aborrero: secgroups: fetch default sg from a tenant using its id, not the name [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/78 [10:32:06] (03approved) 10fnegri: secgroups: fetch default sg from a tenant using its id, not the name [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/78 (owner: 10aborrero) [10:32:13] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10195325 (10jijiki) [10:32:54] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10195326 (10jijiki) [10:32:57] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10195327 (10jijiki) [10:36:49] (03merge) 10aborrero: secgroups: fetch default sg from a tenant using its id, not the name [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/78 [10:37:04] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [10:38:12] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [10:38:15] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [10:52:43] 10wikitech.wikimedia.org: Setting new password at wikitech does not work - https://phabricator.wikimedia.org/T376140#10195371 (10Ladsgroup) 05Open→03Resolved Since you got the password reset working, I'm closing this again. Any further issue probably requires its own ticket. [11:00:53] 10wikitech.wikimedia.org, 06DBA, 07Wikimedia-database-issue, 07Wikimedia-production-error: Wikimedia\Rdbms\DBUnexpectedError: Database servers in cluster30 are overloaded. In order to protect application servers, the circuit breaking to databases of this s... - https://phabricator.wikimedia.org/T376249#10195397 [11:05:00] 10wikitech.wikimedia.org, 06Data Products, 06Data-Engineering, 06DBA: Please drop globalblocks table from labswiki - https://phabricator.wikimedia.org/T375783#10195398 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup I dropped it on master with replication. Let me know if I messed up something. [11:07:54] 10wikitech.wikimedia.org: Setting new password at wikitech does not work - https://phabricator.wikimedia.org/T376140#10195415 (10jijiki) [11:07:55] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10195416 (10jijiki) [11:17:24] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10195436 (10jijiki) [11:17:59] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Cleanup: Wikitech code leftovers - https://phabricator.wikimedia.org/T371378#10195439 (10jijiki) [11:25:54] 06cloud-services-team, 10Phabricator: Create Herald rule for cloud-services-team - https://phabricator.wikimedia.org/T376263#10195465 (10Aklapper) 05Open→03Resolved a:03Aklapper H449 has been created. It does not work retroactively though so if you want to [silently mass-tag](https://www.mediawiki.or... [11:32:30] (03approved) 10sstefanova: global: add mypy checks and fixes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/12 (owner: 10dcaro) [11:32:32] (03update) 10sstefanova: global: add mypy checks and fixes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/12 (owner: 10dcaro) [11:33:21] (03update) 10aborrero: flavors: refactor into the same per-project layout as the rest of the repo [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/76 (https://phabricator.wikimedia.org/T375283) [11:33:28] (03update) 10aborrero: Draft: flavors: refactor into the same per-project layout as the rest of the repo [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/76 (https://phabricator.wikimedia.org/T375283) [11:43:50] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [11:52:00] (03open) 10aborrero: codfw1dev: refresh cloud-flat IPv6 definition [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/79 (https://phabricator.wikimedia.org/T375847) [11:55:02] (03merge) 10aborrero: codfw1dev: refresh cloud-flat IPv6 definition [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/79 (https://phabricator.wikimedia.org/T375847) [11:55:08] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:55:55] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [11:56:17] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:56:51] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [11:58:34] (03open) 10aborrero: codfw1dev: router gw IPv6 port: fix address [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/80 (https://phabricator.wikimedia.org/T375847) [11:59:13] (03merge) 10aborrero: codfw1dev: router gw IPv6 port: fix address [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/80 (https://phabricator.wikimedia.org/T375847) [11:59:22] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:00:49] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:03:11] (03PS3) 10David Caro: deploy: skip sending note to MR if no MR was found [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077336 (https://phabricator.wikimedia.org/T376254) [12:05:29] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10195535 (10Lucas_Werkmeister_WMDE) [12:06:46] (03merge) 10dcaro: global: add mypy checks and fixes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/12 [12:06:46] (03CR) 10CI reject: [V:04-1] deploy: skip sending note to MR if no MR was found [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077336 (https://phabricator.wikimedia.org/T376254) (owner: 10David Caro) [12:08:33] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 [12:08:38] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 [12:19:26] (03PS1) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: don't show apply prompt if plan is noop [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077372 [12:21:18] (03open) 10sstefanova: api: wrap responses inside routes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/13 [12:21:24] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [12:21:47] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [12:21:53] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:22:16] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:22:42] (03CR) 10Arturo Borrero Gonzalez: [C:03+1] "tested with the `test-cookbook` utility, works as expected." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077372 (owner: 10Arturo Borrero Gonzalez) [12:22:54] (03CR) 10CI reject: [V:04-1] wmcs.openstack.tofu: don't show apply prompt if plan is noop [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077372 (owner: 10Arturo Borrero Gonzalez) [12:23:52] (03PS2) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: don't show apply prompt if plan is noop [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077372 [12:23:58] FIRING: SystemdUnitDown: The systemd unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [12:24:02] 06cloud-services-team: SystemdUnitDown Unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been down for long. - https://phabricator.wikimedia.org/T376271 (10phaultfinder) 03NEW [12:27:06] (03CR) 10CI reject: [V:04-1] wmcs.openstack.tofu: don't show apply prompt if plan is noop [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077372 (owner: 10Arturo Borrero Gonzalez) [12:29:13] (03PS3) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: don't show apply prompt if plan is noop [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077372 [12:29:40] (03update) 10sstefanova: api: wrap responses inside routes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/13 [12:32:51] 10wikitech.wikimedia.org, 06DBA, 07Wikimedia-database-issue, 07Wikimedia-production-error: Wikimedia\Rdbms\DBUnexpectedError: Database servers in cluster30 are overloaded. In order to protect application servers, the circuit breaking to databases of this ... - https://phabricator.wikimedia.org/T376249#10195642 [12:35:07] (03approved) 10dcaro: api: wrap responses inside routes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/13 (owner: 10sstefanova) [12:36:38] (03merge) 10sstefanova: api: wrap responses inside routes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/13 [12:36:51] 06cloud-services-team, 10Data-Services: maintain-views fails on labswiki - https://phabricator.wikimedia.org/T375780#10195665 (10fnegri) 05Open→03Resolved a:03fnegri I tested this again after the globalblocks table was dropped from labswik. in T375783. It's now working fine: ` fnegri@an-redacteddb10... [12:38:35] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10195673 (10aborrero) >>! In T375847#10187153, @cmooney wrote: > @aborrero the network assignment is incorrect also. > [[ https://netbo... [12:38:41] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 [12:46:16] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10195699 (10aborrero) I guess next bits to test with neutron would be to enable north-south traffic, meaning working on these two ticke... [12:49:58] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10195719 (10cmooney) >>! In T375847#10195699, @aborrero wrote: > I guess next bits to test with neutron would be to enable north-south... [12:58:24] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: cloudsw: codfw: enable IPv6 - https://phabricator.wikimedia.org/T374713#10195740 (10cmooney) At a high level I think we need to: * Create an aggregate policy on //cloudsw1-b1-codfw// to generate 2a02:ec80:a100::/48 if par... [13:03:09] 10wikitech.wikimedia.org, 06DBA, 07Wikimedia-database-issue, 07Wikimedia-production-error: Wikimedia\Rdbms\DBUnexpectedError: Database servers in cluster30 are overloaded. In order to protect application servers, the circuit breaking to databases of this s... - https://phabricator.wikimedia.org/T376249#10195755 [13:09:49] 06cloud-services-team, 10Phabricator: Create Herald rule for cloud-services-team - https://phabricator.wikimedia.org/T376263#10195791 (10fnegri) Thanks. I will ping you before doing any mass edit! :) [13:17:58] 10wikitech.wikimedia.org, 06DBA, 07Wikimedia-database-issue, 07Wikimedia-production-error: Wikimedia\Rdbms\DBUnexpectedError: Database servers in cluster30 are overloaded. In order to protect application servers, the circuit breaking to databases of this s... - https://phabricator.wikimedia.org/T376249#10195861 [13:23:28] 06cloud-services-team, 10wikitech.wikimedia.org: Reimage eqiad cloudweb hosts to bookworm - https://phabricator.wikimedia.org/T376277 (10taavi) 03NEW [13:27:37] 06cloud-services-team, 10wikitech.wikimedia.org: Reimage eqiad cloudweb hosts to bookworm - https://phabricator.wikimedia.org/T376277#10195916 (10taavi) [13:27:40] 06cloud-services-team: Complete upgrading WMCS bare metal hosts from Bullseye to Bookworm - https://phabricator.wikimedia.org/T375217#10195917 (10taavi) [13:39:18] 10wikitech.wikimedia.org, 06DBA, 07Wikimedia-database-issue, 07Wikimedia-production-error: Wikimedia\Rdbms\DBUnexpectedError: Database servers in cluster30 are overloaded. In order to protect application servers, the circuit breaking to databases of this ... - https://phabricator.wikimedia.org/T376249#10195968 [13:49:07] 10wikitech.wikimedia.org: OAuth consumers registered locally at Wikitech are no longer configured to be used - https://phabricator.wikimedia.org/T376188#10196015 (10Tgr) Could just copy the relevant database rows, there isn't anything wiki-specific in them (other than the user ID). Although if there are only twe... [13:56:12] !log fnegri@cloudcumin1001 catalyst-dev START - Cookbook wmcs.vps.add_user_to_project for user 'kindrobot' in role 'member' (T376211) [13:56:14] fnegri@cloudcumin1001: Unknown project "catalyst-dev" [13:56:14] T376211: Request creation of catalyst-dev VPS project - https://phabricator.wikimedia.org/T376211 [13:56:18] !log fnegri@cloudcumin1001 catalyst-dev END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'kindrobot' in role 'member' (T376211) [13:56:18] fnegri@cloudcumin1001: Unknown project "catalyst-dev" [13:56:35] !log fnegri@cloudcumin1001 catalyst-dev START - Cookbook wmcs.vps.add_user_to_project for user 'ebomani' in role 'member' (T376211) [13:56:35] fnegri@cloudcumin1001: Unknown project "catalyst-dev" [13:56:41] !log fnegri@cloudcumin1001 catalyst-dev END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'ebomani' in role 'member' (T376211) [13:56:41] fnegri@cloudcumin1001: Unknown project "catalyst-dev" [13:58:39] 10Cloud-VPS (Project-requests): Request creation of catalyst-dev VPS project - https://phabricator.wikimedia.org/T376211#10196039 (10fnegri) 05In progress→03Resolved The project was created, and users `kindrobot` and `ebomani` added with role "member". [13:59:32] 10cloud-services-team (FY2024/2025-Q1-Q2): Test using phabricator-maintenance-bot to sync wmcs-related boards - https://phabricator.wikimedia.org/T358251#10196042 (10fnegri) [14:05:43] (03open) 10dcaro: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [14:06:53] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [14:10:48] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [14:14:05] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [14:19:14] 10wikitech.wikimedia.org: Toolforge Image Bot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376223#10196135 (10taavi) 05Open→03Resolved [14:19:55] (03PS4) 10David Caro: deploy: skip sending note to MR if no MR was found [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077336 (https://phabricator.wikimedia.org/T376254) [14:19:56] (03PS6) 10David Caro: ceph: use prometheus-node-pinger for ping checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076978 [14:19:56] (03PS8) 10David Caro: toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 [14:21:49] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [14:23:21] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 (owner: 10David Caro) [14:27:13] (03CR) 10David Caro: [C:03+2] deploy: skip sending note to MR if no MR was found [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077336 (https://phabricator.wikimedia.org/T376254) (owner: 10David Caro) [14:34:36] (03Merged) 10jenkins-bot: deploy: skip sending note to MR if no MR was found [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077336 (https://phabricator.wikimedia.org/T376254) (owner: 10David Caro) [14:35:07] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [14:36:51] (03PS1) 10JMeybohm: kubernetes::worker_containerd: Fix registry_auth hiera key [labs/private] - 10https://gerrit.wikimedia.org/r/1077404 (https://phabricator.wikimedia.org/T362408) [14:39:58] (03update) 10taavi: Call accounts by their proper names [toolforge-repos/techcontribs] - 10https://gitlab.wikimedia.org/toolforge-repos/techcontribs/-/merge_requests/1 [14:40:47] (03update) 10taavi: Call accounts by their proper names [toolforge-repos/techcontribs] - 10https://gitlab.wikimedia.org/toolforge-repos/techcontribs/-/merge_requests/1 [14:41:24] (03CR) 10JMeybohm: [V:03+2 C:03+2] kubernetes::worker_containerd: Fix registry_auth hiera key [labs/private] - 10https://gerrit.wikimedia.org/r/1077404 (https://phabricator.wikimedia.org/T362408) (owner: 10JMeybohm) [14:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:58:51] (03open) 10aborrero: projects: remove id attribute [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/81 [15:02:27] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: introduce additional gitlab-ci automation - https://phabricator.wikimedia.org/T370652#10196276 (10aborrero) See: https://gitlab.wikimedia.org/bd808/deployment-prep-opentofu [15:05:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:06:13] (03open) 10sstefanova: api: wrap errors in ApiResponse [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/15 [15:08:39] (03merge) 10chlod: Call accounts by their proper names [toolforge-repos/techcontribs] - 10https://gitlab.wikimedia.org/toolforge-repos/techcontribs/-/merge_requests/1 (owner: 10taavi) [15:09:47] (03CR) 10Arturo Borrero Gonzalez: [C:03+1] "LGTM." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076978 (owner: 10David Caro) [15:10:04] (03update) 10aborrero: projects: remove id attribute [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/81 [15:10:42] (03PS9) 10David Caro: toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 [15:12:54] (03CR) 10FNegri: [C:03+1] wmcs.openstack.tofu: don't show apply prompt if plan is noop [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077372 (owner: 10Arturo Borrero Gonzalez) [15:13:16] (03CR) 10Arturo Borrero Gonzalez: [C:03+2] wmcs.openstack.tofu: don't show apply prompt if plan is noop [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077372 (owner: 10Arturo Borrero Gonzalez) [15:14:24] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 (owner: 10David Caro) [15:15:06] (03CR) 10FNegri: wmcs.openstack.tofu: don't show apply prompt if plan is noop (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077372 (owner: 10Arturo Borrero Gonzalez) [15:15:25] (03CR) 10David Caro: [C:03+1] "LGTM, feel free to ignore the nit" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1077372 (owner: 10Arturo Borrero Gonzalez) [15:15:56] (03CR) 10David Caro: [C:03+2] ceph: use prometheus-node-pinger for ping checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076978 (owner: 10David Caro) [15:16:53] (03PS10) 10David Caro: toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 [15:19:06] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: introduce additional gitlab-ci automation - https://phabricator.wikimedia.org/T370652#10196313 (10aborrero) >>! In T370652#10196276, @aborrero wrote: > See: > > * https://gitlab.wikimedia.org/bd808/deployment-prep-opentofu > * https://gitlab.wikimedia.... [15:19:23] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for Antonin Delpeuch (Pintoch) - https://phabricator.wikimedia.org/T374995#10196314 (10bd808) The next step on https://wikitech.wikimedia.org/wiki/Volunteer_NDA seems to be "After that, ask in the Phabricator task to make you a member of the #w... [15:19:36] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for SD0001 - https://phabricator.wikimedia.org/T374998#10196315 (10bd808) The next step on https://wikitech.wikimedia.org/wiki/Volunteer_NDA seems to be "After that, ask in the Phabricator task to make you a member of the #wmf-nda project." Ple... [15:20:01] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for Lucas Werkmeister - https://phabricator.wikimedia.org/T375001#10196319 (10bd808) The next step on https://wikitech.wikimedia.org/wiki/Volunteer_NDA seems to be "After that, ask in the Phabricator task to make you a member of the #wmf-nda pr... [15:20:17] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for TheProtonade - https://phabricator.wikimedia.org/T375007#10196321 (10bd808) The next step on https://wikitech.wikimedia.org/wiki/Volunteer_NDA seems to be "After that, ask in the Phabricator task to make you a member of the #wmf-nda project... [15:20:33] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for JJMC89 - https://phabricator.wikimedia.org/T375041#10196323 (10bd808) The next step on https://wikitech.wikimedia.org/wiki/Volunteer_NDA seems to be "After that, ask in the Phabricator task to make you a member of the #wmf-nda project." Ple... [15:20:51] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for Waldir Pimenta (Waldyrious) - https://phabricator.wikimedia.org/T375110#10196324 (10bd808) The next step on https://wikitech.wikimedia.org/wiki/Volunteer_NDA seems to be "After that, ask in the Phabricator task to make you a member of the #... [15:22:31] (03open) 10dcaro: api-gateway: enable components-api on local [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/545 [15:23:22] (03merge) 10aborrero: projects: remove id attribute [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/81 [15:23:33] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:23:34] !log aborrero@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch [15:23:48] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:24:15] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [15:25:00] (03CR) 10FNegri: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076978 (owner: 10David Caro) [15:26:01] (03open) 10aborrero: projects: create 'test-project-creation-delete-me' project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/82 [15:35:55] (03Merged) 10jenkins-bot: ceph: use prometheus-node-pinger for ping checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076978 (owner: 10David Caro) [15:36:46] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10196424 (10Raymond_Ndibe) [15:37:00] 06cloud-services-team, 10Toolforge (Toolforge iteration 15): component.deploy cookbook fails for branch "main" - https://phabricator.wikimedia.org/T376254#10196437 (10dcaro) a:03dcaro [15:37:02] 06cloud-services-team, 10Toolforge (Toolforge iteration 15): component.deploy cookbook fails for branch "main" - https://phabricator.wikimedia.org/T376254#10196438 (10dcaro) p:05Triage→03Medium [15:37:10] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10196441 (10Raymond_Ndibe) [15:37:56] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10196442 (10Raymond_Ndibe) [15:39:03] 06cloud-services-team, 10Toolforge (Toolforge iteration 15): component.deploy cookbook fails for branch "main" - https://phabricator.wikimedia.org/T376254#10196440 (10dcaro) 05Open→03Resolved [15:39:10] 06cloud-services-team, 10Toolforge (Toolforge iteration 15): component.deploy cookbook fails for branch "main" - https://phabricator.wikimedia.org/T376254#10196435 (10dcaro) Done :) [15:39:57] (03update) 10aborrero: projects: create 'test-project-creation-delete-me' project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/82 [15:40:39] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10196466 (10Raymond_Ndibe) [15:43:38] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [15:44:13] 10wikitech.wikimedia.org: OAuth consumers registered locally at Wikitech are no longer configured to be used - https://phabricator.wikimedia.org/T376188#10196522 (10bd808) >>! In T376188#10196015, @Tgr wrote: > Could just copy the relevant database rows, there isn't anything wiki-specific in them (other than the... [15:48:33] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: cloudsw: codfw: enable IPv6 - https://phabricator.wikimedia.org/T374713#10196558 (10aborrero) here is a proposal: * 2a02:ec80:a100:fe01::/64 - cr1-codfw uplink * 2a02:ec80:a100:fe02::/64 - cr2-codfw uplink * 2a02:ec80:a10... [15:49:38] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [15:51:11] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [15:51:18] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] (slavina/wrap-all-responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [15:55:02] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] (slavina/wrap-all-responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [16:06:26] FIRING: SystemdUnitDown: The service unit rsync_enterprise_htmldumps.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:07:03] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] (slavina/wrap-all-responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [16:18:54] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications, 07Wikimedia-production-error: Wikitech notifications failing to load cross-wiki - https://phabricator.wikimedia.org/T376305 (10taavi) 03NEW [16:21:42] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] (slavina/wrap-all-responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [16:26:41] FIRING: SystemdUnitDown: The systemd unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:28:31] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] (slavina/wrap-all-responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [16:29:59] 06cloud-services-team, 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10196898 (10taavi) Fwiw, the sync is now broken since https://wikitech.wikimedia.org/dumps/ is no longer served fr... [16:31:07] (03approved) 10fnegri: projects: create 'test-project-creation-delete-me' project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/82 (owner: 10aborrero) [16:42:03] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] (slavina/wrap-all-responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [16:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:05:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:06:07] (03PS1) 10Majavah: auth: Properly remove OATHAuth support [labs/striker] - 10https://gerrit.wikimedia.org/r/1077444 (https://phabricator.wikimedia.org/T373461) [17:06:17] (03PS2) 10Majavah: auth: Properly remove 2FA support [labs/striker] - 10https://gerrit.wikimedia.org/r/1077444 (https://phabricator.wikimedia.org/T373461) [17:08:47] (03CR) 10CI reject: [V:04-1] auth: Properly remove 2FA support [labs/striker] - 10https://gerrit.wikimedia.org/r/1077444 (https://phabricator.wikimedia.org/T373461) (owner: 10Majavah) [17:10:10] (03PS3) 10Majavah: auth: Properly remove 2FA support [labs/striker] - 10https://gerrit.wikimedia.org/r/1077444 (https://phabricator.wikimedia.org/T373461) [17:27:02] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge (Toolforge iteration 15): Decision Request: To strictly enforce semantic versioning rules for toolforge services' APIs or not - https://phabricator.wikimedia.org/T373072#10197224 (10Raymond_Ndibe) [17:27:51] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge (Toolforge iteration 15): Decision Request: To strictly enforce semantic versioning rules for toolforge services' APIs or not - https://phabricator.wikimedia.org/T373072#10197236 (10Raymond_Ndibe) [17:34:26] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15): [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10197263 (10Raymond_Ndibe) [17:34:55] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge (Toolforge iteration 15): Decision Request: To strictly enforce semantic versioning rules for toolforge services' APIs or not - https://phabricator.wikimedia.org/T373072#10197261 (10Raymond_Ndibe) 05In progress→03Resolved [18:16:44] (03update) 10sstefanova: api-gateway: enable components-api on local [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/545 (owner: 10dcaro) [18:16:44] (03approved) 10sstefanova: api-gateway: enable components-api on local [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/545 (owner: 10dcaro) [18:58:32] 10wikitech.wikimedia.org: OAuth consumers registered locally at Wikitech are no longer configured to be used - https://phabricator.wikimedia.org/T376188#10197509 (10Umherirrender) Maybe empty the local oauth tables when no longer used to avoid orphaned rows (maybe as part of {T376129}) [19:07:35] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10197547 (10hashar) If the W... [19:12:41] 10Tool-incunabula, 06Community-Tech, 10Wikimedia OCR: OCR should be able to return text position information - https://phabricator.wikimedia.org/T376331 (10Marnanel) 03NEW [20:22:56] (03CR) 10BryanDavis: [V:03+1 C:03+2] "Diffs look good. `git grep -il oath` has no hits with the patch applied. My dev environment ran fine with the change as well (after resurr" [labs/striker] - 10https://gerrit.wikimedia.org/r/1077444 (https://phabricator.wikimedia.org/T373461) (owner: 10Majavah) [20:24:10] (03Merged) 10jenkins-bot: auth: Properly remove 2FA support [labs/striker] - 10https://gerrit.wikimedia.org/r/1077444 (https://phabricator.wikimedia.org/T373461) (owner: 10Majavah) [20:26:41] FIRING: SystemdUnitDown: The systemd unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [20:29:22] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10197775 (10cscott) I'm seei... [20:29:36] (03PS1) 10BryanDavis: dev: Remove legacy docker-compose `version` values [labs/striker] - 10https://gerrit.wikimedia.org/r/1077473 [20:29:36] (03PS1) 10BryanDavis: dev: Allow insecure apt actions in Keystone and MediaWiki Docker [labs/striker] - 10https://gerrit.wikimedia.org/r/1077474 [20:44:25] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10197799 (10bd808) >>! In T3... [20:48:53] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for Waldir Pimenta (Waldyrious) - https://phabricator.wikimedia.org/T375110#10197817 (10waldyrious) Thanks for the follow-up, @bd808! Is there someone we should ping for that request? The instructions are not clear in that respect. [20:58:48] 10Toolforge (Quota-requests): Request increased quota for Toolforge tool - https://phabricator.wikimedia.org/T376338 (10Mmarx) 03NEW [20:59:00] 10Toolforge (Quota-requests): Request increased quota for sqid Toolforge tool - https://phabricator.wikimedia.org/T376338#10197865 (10Mmarx) [21:21:37] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10197977 (10bd808) I found a... [21:29:43] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10198022 (10cscott) Can we s... [21:33:20] (03CR) 10BryanDavis: [C:03+2] dev: Remove legacy docker-compose `version` values [labs/striker] - 10https://gerrit.wikimedia.org/r/1077473 (owner: 10BryanDavis) [21:34:34] (03Merged) 10jenkins-bot: dev: Remove legacy docker-compose `version` values [labs/striker] - 10https://gerrit.wikimedia.org/r/1077473 (owner: 10BryanDavis) [21:34:45] (03CR) 10BryanDavis: "This is probably the worst way to fix my local dev environment, but it did fix it. I won't be sad if Taavi tell me to never merge this mes" [labs/striker] - 10https://gerrit.wikimedia.org/r/1077474 (owner: 10BryanDavis) [21:56:08] 10Cloud-VPS (Project-requests), 07affects-Miraheze: Request creation of createwikitest VPS project - https://phabricator.wikimedia.org/T375454#10198063 (10Aklapper) We generally do not grant Cloud VPS projects for single user development use (basically "laptop in the cloud"). Wikis could be hosted if their... [22:00:21] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for Waldir Pimenta (Waldyrious) - https://phabricator.wikimedia.org/T375110#10198085 (10bd808) >>! In T375110#10197817, @waldyrious wrote: > Thanks for the follow-up, @bd808! Is there someone we should ping for that request? The instructions ar... [22:28:39] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: cloudsw: codfw: enable IPv6 - https://phabricator.wikimedia.org/T374713#10198161 (10cmooney) That seems fine to me @aborrero thanks! [22:41:15] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10198217 (10bd808) Failures... [22:42:59] 10Cloud-Services: Prepare "What's new with Wikimedia Cloud Services" presentation for WikiConNA 2024 - https://phabricator.wikimedia.org/T373159#10198218 (10bd808) 05Open→03Resolved https://commons.wikimedia.org/wiki/File:What%27s_new_with_Wikimedia_Cloud_Services,_WikiConNA_2024.pdf [22:47:30] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in 22d 23h 58m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [23:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks