[01:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:04:35] 10Cloud-VPS, 10Wikispore: Vanity domain for wikinyc.wmcloud.org (wikispore project) - https://phabricator.wikimedia.org/T368236#10302638 (10Samwilson) Oh right, that makes sense. I've updated the title and description. [02:06:52] 10Cloud-VPS, 10Wikispore: Vanity domain for wikinyc.wmcloud.org (wikispore project) - https://phabricator.wikimedia.org/T368236#10302636 (10Samwilson) [04:35:29] 06Toolforge-standards-committee: Adoption request for jawi - https://phabricator.wikimedia.org/T379340 (10Hakimi97) 03NEW [04:36:23] 06Toolforge-standards-committee: Adoption request for jawi - https://phabricator.wikimedia.org/T379340#10302715 (10Hakimi97) [06:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:40:09] 06cloud-services-team, 10Toolforge: [builds-builder] Cache .m2 folder (local maven repository) between builds - https://phabricator.wikimedia.org/T350307#10302818 (10Slst2020) 05Open→03Resolved a:03Slst2020 That's great to hear, thank you! [09:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:52:14] (03approved) 10sstefanova: openapi: add external url setting [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/52 (owner: 10dcaro) [10:05:17] (03approved) 10sstefanova: auth: allow pass through for deploy urls with tokens [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/51 (https://phabricator.wikimedia.org/T362066) (owner: 10dcaro) [10:13:01] (03update) 10sstefanova: add token validation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/32 (https://phabricator.wikimedia.org/T362066) (owner: 10dcaro) [10:27:59] 14Toolforge (Toolforge iteration 09): [infra] Add alert when workers have a sustained large amount of D processes - https://phabricator.wikimedia.org/T362093#10303249 (10Leloiandudu) @dcaro just curious, what happens when the alert goes off? do you manually go and kill the worker? [10:33:53] (03update) 10sstefanova: deploy: add delete endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T379093) [10:37:21] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 07Epic: [Hypotesis] 6.3.5 Develop the sustainability score - https://phabricator.wikimedia.org/T376896#10303272 (10Slst2020) 05Stalled→03In progress [10:37:42] 10Toolforge (Toolforge iteration 16): [components-api] add endpoints to list deployments - https://phabricator.wikimedia.org/T379350 (10Slst2020) 03NEW [10:39:38] 10Toolforge (Toolforge iteration 16): [components-api] add endpoints to list deployments - https://phabricator.wikimedia.org/T379350#10303293 (10Slst2020) 05Open→03In progress [10:41:05] 06cloud-services-team, 06DC-Ops, 06Infrastructure-Foundations: kernel message: SGX disabled by BIOS - https://phabricator.wikimedia.org/T379351 (10fnegri) 03NEW [10:44:04] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: 2024-09-21 NodeDown cloudvirt1063 - https://phabricator.wikimedia.org/T375223#10303311 (10fnegri) 05In progress→03Resolved I've created {T379351} to discuss what to do about the kernel message. This task is now completed. [10:53:40] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: IPv6 support in cloud-private - https://phabricator.wikimedia.org/T379283#10303339 (10aborrero) p:05Triage→03Medium [10:56:05] 06cloud-services-team, 10Cloud-VPS, 07IPv6: IPv6 for cloud-realm services - https://phabricator.wikimedia.org/T379282#10303354 (10aborrero) p:05Triage→03Medium [10:59:57] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: IPv6 support in cloud-private - https://phabricator.wikimedia.org/T379283#10303376 (10aborrero) My feeling is that https://netbox.wikimedia.org/ipam/prefixes/1089/ `2a02:ec80:a100:200::/56` which is `wmcs cloud-private codfw (non-openstack)` w... [11:34:58] 06cloud-services-team, 06DC-Ops, 06Infrastructure-Foundations: kernel message: SGX disabled by BIOS - https://phabricator.wikimedia.org/T379351#10303416 (10MoritzMuehlenhoff) We don't use or need SGX for virtualisation servers. It's a feature invented by Intel (AMD never adopted it, which is telling by itsel... [11:58:22] (03open) 10sstefanova: deployments: add list endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T379093) [12:05:59] 06cloud-services-team, 06DC-Ops, 06Infrastructure-Foundations: kernel message: SGX disabled by BIOS - https://phabricator.wikimedia.org/T379351#10303474 (10fnegri) @MoritzMuehlenhoff thanks! Do you suggest to keep it disabled and ignore the kernel message? Do you know why the kernel message is only shown on... [12:24:11] (03update) 10sstefanova: deployments: add list endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T379093) [12:40:53] 06cloud-services-team, 10Cloud-VPS: openstack: nova-fullstack: add support for IPv6 - https://phabricator.wikimedia.org/T379356 (10aborrero) 03NEW [12:40:57] 06cloud-services-team, 10Cloud-VPS: openstack: nova-fullstack: add support for IPv6 - https://phabricator.wikimedia.org/T379356#10303553 (10aborrero) p:05Triage→03Medium [12:51:46] (03update) 10sstefanova: deployments: add list endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T379093) [13:01:38] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [components-api] Develop the webhook mechanism to trigger a deployment - https://phabricator.wikimedia.org/T362066#10303587 (10dcaro) 05In progress→03Resolved [13:26:58] (03update) 10sstefanova: deployments: add list endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T379093) [13:29:56] (03update) 10sstefanova: deployments: add list endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T379093 https://phabricator.wikimedia.org/T379350) [13:32:34] !log aborrero@cloudcumin2001 admin START - Cookbook wmcs.openstack.restart_openstack [13:33:36] !log aborrero@cloudcumin2001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [13:44:42] !log aborrero@cloudcumin2001 admin START - Cookbook wmcs.openstack.restart_openstack [13:45:30] !log aborrero@cloudcumin2001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [13:52:40] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate deployment-poolcounter06.deployment-prep.eqiad.wmflabs is about to expire in 19d 23h 58m 30s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [14:04:23] 06cloud-services-team, 10Cloud-VPS: openstack: wmf sink: extend it to support IPv6 - https://phabricator.wikimedia.org/T378192#10303807 (10aborrero) I saw several failures today. I was unable to create VMs by hand, because nova failures when scheduling VMs. Also, some failures related to nova-fullstack {T3793... [14:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:21:05] 10cloud-services-team (FY2024/2025-Q1-Q2): Kernel alerts disappear too quickly - https://phabricator.wikimedia.org/T379378 (10fnegri) 03NEW [15:22:02] 10cloud-services-team (FY2024/2025-Q1-Q2): Kernel alerts disappear too quickly - https://phabricator.wikimedia.org/T379378#10304251 (10fnegri) 05Open→03In progress p:05Triage→03Medium a:03fnegri [15:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:42:05] !log aborrero@cloudcumin2001 admin START - Cookbook wmcs.openstack.restart_openstack [16:42:51] !log aborrero@cloudcumin2001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [16:45:59] !log aborrero@cloudcumin2001 admin START - Cookbook wmcs.openstack.restart_openstack [16:47:30] !log aborrero@cloudcumin2001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [16:53:19] (03PS1) 10Arturo Borrero Gonzalez: wmcs.openstack.restart_openstack: add runtime_description() [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1088597 [16:56:40] (03CR) 10CI reject: [V:04-1] wmcs.openstack.restart_openstack: add runtime_description() [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1088597 (owner: 10Arturo Borrero Gonzalez) [16:56:54] 06cloud-services-team, 10Cloud-VPS: openstack: wmf sink: extend it to support IPv6 - https://phabricator.wikimedia.org/T378192#10304715 (10aborrero) ok, after restarting a few more openstack things: `lang=shell-session aborrero@bastion-codfw1dev-04:~$ host please-work.cloudinfra-codfw1dev.codfw1dev.wikimedia.... [16:57:37] (03PS2) 10Arturo Borrero Gonzalez: wmcs.openstack.restart_openstack: add runtime_description() [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1088597 [17:00:51] (03CR) 10CI reject: [V:04-1] wmcs.openstack.restart_openstack: add runtime_description() [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1088597 (owner: 10Arturo Borrero Gonzalez) [17:01:45] 06Toolforge-standards-committee: Adoption request for jawi - https://phabricator.wikimedia.org/T379340#10304722 (10JJMC89) 05Open→03Declined The tool's software is not in compliance with the Cloud Services Terms of use as it is not [[https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_us... [17:03:11] (03PS3) 10Arturo Borrero Gonzalez: wmcs.openstack.restart_openstack: add runtime_description() [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1088597 [17:05:17] 06cloud-services-team, 10Cloud-VPS: openstack: wmf sink: extend it to support IPv6 - https://phabricator.wikimedia.org/T378192#10304726 (10aborrero) 05In progress→03Resolved a:03aborrero [17:33:33] 06Toolforge-standards-committee: Adoption request for jawi - https://phabricator.wikimedia.org/T379340#10304795 (10bd808) For clarity, this tool's source code has no obvious declared license. This puts the code in the US default copyright status of the lifetime of the author plus 70 years after their death.... [17:42:49] !log fnegri@cloudcumin1001 wmde-techwishes-survey START - Cookbook wmcs.vps.add_user_to_project for user 'awight' in role 'member' (T378975) [17:42:51] fnegri@cloudcumin1001: Unknown project "wmde-techwishes-survey" [17:42:51] T378975: Request creation of wmde-techwishes-survey VPS project - https://phabricator.wikimedia.org/T378975 [17:42:55] !log fnegri@cloudcumin1001 wmde-techwishes-survey END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'awight' in role 'member' (T378975) [17:42:56] fnegri@cloudcumin1001: Unknown project "wmde-techwishes-survey" [17:43:04] !log fnegri@cloudcumin1001 wmde-techwishes-survey START - Cookbook wmcs.vps.add_user_to_project for user 'wmde-fisch' in role 'member' (T378975) [17:43:04] fnegri@cloudcumin1001: Unknown project "wmde-techwishes-survey" [17:43:10] !log fnegri@cloudcumin1001 wmde-techwishes-survey END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'wmde-fisch' in role 'member' (T378975) [17:43:10] fnegri@cloudcumin1001: Unknown project "wmde-techwishes-survey" [17:46:01] 10Cloud-VPS (Project-requests), 10WMDE-TechWish-Survey, 07Unplanned-Sprint-Work, 03WMDE-TechWish-Sprint-2024-10-16: Request creation of wmde-techwishes-survey VPS project - https://phabricator.wikimedia.org/T378975#10304814 (10fnegri) 05Open→03Resolved a:03fnegri The project was created and users... [17:50:01] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10304857 (10CDobbins) |**Wikitech account/LDAP:**| cdobbins | |**SUL account**| CDobbins-WMF | |**Account linked on IDM** |Y| |**I have read [[ https://wikitech.wikimedia.org/wiki/MediaWiki:L... [18:07:18] 10PAWS, 07Upstream: PAWS kills active users servers that are not connected to a user session - https://phabricator.wikimedia.org/T188684#10304905 (10rook) I've tried to spread out cluster rebuilds some since my last comment. Haven't heard similar issues since then so that may well have been the issue. Please r... [18:07:24] 10PAWS, 07Upstream: PAWS kills active users servers that are not connected to a user session - https://phabricator.wikimedia.org/T188684#10304906 (10rook) 05Open→03Resolved [18:14:37] 10wikitech.wikimedia.org: Wikitech users being unexpectedly prompted for 2FA tokens - https://phabricator.wikimedia.org/T379398 (10bd808) 03NEW [18:15:32] 10PAWS: Upgrade jupyter chart - https://phabricator.wikimedia.org/T379400 (10rook) 03NEW [18:15:38] 10PAWS: Upgrade jupyter chart - https://phabricator.wikimedia.org/T379400#10304973 (10rook) 05Open→03Stalled [18:20:29] 10wikitech.wikimedia.org: Wikitech users being unexpectedly prompted for 2FA tokens - https://phabricator.wikimedia.org/T379398#10304997 (10bd808) I just tested login from an incognito window with my [[https://wikitech.wikimedia.org/wiki/User:BryanDavis|wikitech:User:BryanDavis]] account and it is reproducing th... [18:22:35] 10wikitech.wikimedia.org: Wikitech users being unexpectedly prompted for 2FA tokens - https://phabricator.wikimedia.org/T379398#10305005 (10bd808) I wonder if the easiest fix would be to rename (or archive and drop) the legacy `oathauth_devices`, `oathauth_types`, `oathauth_users`, and `oathauth_users_restore` t... [18:23:04] 10wikitech.wikimedia.org, 10MediaWiki-extensions-OATHAuth: Wikitech users being unexpectedly prompted for 2FA tokens - https://phabricator.wikimedia.org/T379398#10305006 (10bd808) [18:25:34] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10305016 (10bd808) [18:40:58] 06Toolforge-standards-committee: Adoption request for jawi - https://phabricator.wikimedia.org/T379340#10305075 (10LucasWerkmeister) If you want to reimplement the tool (my very naive guess, without ever having seen the tool in action, is that it seems like a manageable task in principle – the existing sourc... [18:41:32] 10wikitech.wikimedia.org, 10MediaWiki-extensions-OATHAuth: Wikitech users being unexpectedly prompted for 2FA tokens - https://phabricator.wikimedia.org/T379398#10305078 (10bd808) The first report of this issue was T376267#10292062 on 2024-11-05. The first log event for that user is timestamped Nov 5, 2024 @ 1... [19:08:46] 06cloud-services-team, 10Toolforge: mysqldump Command Not Found on tools Bastion - https://phabricator.wikimedia.org/T378882#10305208 (10LucasWerkmeister) 05Resolved→03Open Reopening, as I’d respectfully ask you to reconsider installing mysqldump on the bastion directly. `toolforge jobs` does not support s... [21:17:56] 06cloud-services-team, 10Cloud-VPS: Enable use of web proxy for wikiwho.net domain - https://phabricator.wikimedia.org/T376637#10305615 (10MusikAnimal) If the IP is stable, then I think that's all they need. I'm still waiting to hear back (it's a bit slow as I'm talking to an engineer who then relays the messa... [22:32:31] 10wikitech.wikimedia.org, 10MediaWiki-extensions-OATHAuth: Wikitech users being unexpectedly prompted for 2FA tokens - https://phabricator.wikimedia.org/T379398#10305824 (10Reedy) >>! In T379398#10305005, @bd808 wrote: > I wonder if the easiest fix would be to rename (or archive and drop) the legacy `oathauth_... [23:00:15] 10wikitech.wikimedia.org, 10MediaWiki-extensions-OATHAuth: Wikitech users being unexpectedly prompted for 2FA tokens - https://phabricator.wikimedia.org/T379398#10305902 (10bd808) `lang=php $ mw-debug-repl labswiki Psy Shell v0.12.3 (PHP 7.4.33 — cli) by Justin Hileman > $wmgUseOATHAuth = true > $wmgUseCentra... [23:00:37] 10wikitech.wikimedia.org, 10MediaWiki-extensions-OATHAuth: Wikitech users being unexpectedly prompted for 2FA tokens - https://phabricator.wikimedia.org/T379398#10305903 (10Reedy) If those accounts aren't attached... `lang=php public function centralIdFromLocalUser( UserIdentity $user, $audience = self::AU... [23:09:23] 10wikitech.wikimedia.org, 10MediaWiki-extensions-OATHAuth: Wikitech users being unexpectedly prompted for 2FA tokens - https://phabricator.wikimedia.org/T379398#10305910 (10Reedy) https://meta.wikimedia.org/wiki/Special:CentralAuth?target=Robert%20Timm%20(WMDE) isn't attached https://meta.wikimedia.org/wiki/S... [23:37:39] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10305942 (10Reedy) >>! In T376267#10304857, @CDobbins wrote: > |**Wikitech account/LDAP:**| cdobbins | > |**SUL account**| CDobbins-WMF | > |**Account linked on IDM** |Y| > |**I have read [[... [23:53:44] 10wikitech.wikimedia.org, 10MediaWiki-extensions-OATHAuth: Wikitech users being unexpectedly prompted for 2FA tokens - https://phabricator.wikimedia.org/T379398#10305975 (10Reedy) I didn't test it before I deleted that row, but... ` > $roti = $userFactory->newFromName( "Robert Timm (WMDE)" ); = MediaWiki\User... [23:57:56] 10wikitech.wikimedia.org, 10MediaWiki-extensions-OATHAuth: Wikitech users being unexpectedly prompted for 2FA tokens - https://phabricator.wikimedia.org/T379398#10305978 (10Reedy) And when I reinserted it for testing... ` > $totp = $authUser->getModule(); = MediaWiki\Extension\OATHAuth\Module\TOTP {#8352} >...