[03:45:21] 10Cloud-VPS (Quota-requests): Increase CPUs for webperformancetest - https://phabricator.wikimedia.org/T379514 (10Peter) 03NEW [03:45:32] 10Cloud-VPS (Quota-requests): Increase CPUs for webperformancetest - https://phabricator.wikimedia.org/T379514#10308109 (10Peter) [08:06:26] 06Toolforge-standards-committee: Adoption request for jawi - https://phabricator.wikimedia.org/T379340#10308341 (10Hakimi97) [08:51:29] 06Toolforge-standards-committee: Adoption request for jawi - https://phabricator.wikimedia.org/T379340#10308487 (10Hakimi97) [09:03:18] (03open) 10samwilson: Add 'refresh' button to clear cache and re-request all data [toolforge-repos/wdlocator] - 10https://gitlab.wikimedia.org/toolforge-repos/wdlocator/-/merge_requests/31 (https://phabricator.wikimedia.org/T364287) [09:10:53] 06cloud-services-team, 10Tools, 10Pywikibot, 10Pywikibot-Scripts: Implement a webservice at toolforge.org based on create_isbn_edition script - https://phabricator.wikimedia.org/T379488#10308509 (10dcaro) [09:24:06] (03approved) 10dcaro: [toolforge-weld] refactor parse_quantity [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/64 (https://phabricator.wikimedia.org/T361120) (owner: 10raymond-ndibe) [09:25:41] (03update) 10dcaro: [jobs-api] refactor validate_kube_quant [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/118 (https://phabricator.wikimedia.org/T361120) (owner: 10raymond-ndibe) [09:26:53] (03update) 10fnegri: Warn about unknown keys in service.template [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/59 (owner: 10lucaswerkmeister) [09:28:06] (03merge) 10samwilson: Add 'refresh' button to clear cache and re-request all data [toolforge-repos/wdlocator] - 10https://gitlab.wikimedia.org/toolforge-repos/wdlocator/-/merge_requests/31 (https://phabricator.wikimedia.org/T364287) [09:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:01:59] (03open) 10samwilson: More refreshing cache-skipping [toolforge-repos/wdlocator] - 10https://gitlab.wikimedia.org/toolforge-repos/wdlocator/-/merge_requests/32 [10:04:50] (03merge) 10samwilson: More refreshing cache-skipping [toolforge-repos/wdlocator] - 10https://gitlab.wikimedia.org/toolforge-repos/wdlocator/-/merge_requests/32 [10:19:56] (03update) 10dcaro: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] (allow_getting_config_from_env) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225) (owner: 10raymond-ndibe) [10:20:30] (03update) 10dcaro: [maintain-harbor] do not clean up images currently running in production [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/35 (https://phabricator.wikimedia.org/T377854) (owner: 10raymond-ndibe) [10:20:52] (03update) 10dcaro: [maintain-harbor] specify config through environment variables [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/36 (https://phabricator.wikimedia.org/T358225) (owner: 10raymond-ndibe) [10:22:09] 06cloud-services-team, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: kernel message: SGX disabled by BIOS - https://phabricator.wikimedia.org/T379351#10308928 (10fnegri) > Do you know why the kernel message is only shown on this host? It's actually present on other hosts as well, I was testi... [10:44:45] 10Cloud-VPS (Quota-requests): Increase CPUs for webperformancetest - https://phabricator.wikimedia.org/T379514#10309010 (10fnegri) a:03fnegri [10:51:50] (03approved) 10dcaro: deploy: add delete endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T379093) (owner: 10sstefanova) [10:51:56] (03update) 10dcaro: deploy: add delete endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T379093) (owner: 10sstefanova) [10:52:42] !log fnegri@cloudcumin1001 webperformancetest START - Cookbook wmcs.openstack.quota_increase (T379514) [10:52:43] fnegri@cloudcumin1001: Unknown project "webperformancetest" [10:52:43] T379514: Increase CPUs for webperformancetest - https://phabricator.wikimedia.org/T379514 [10:52:50] !log fnegri@cloudcumin1001 webperformancetest END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T379514) [10:52:50] fnegri@cloudcumin1001: Unknown project "webperformancetest" [10:53:24] (03update) 10dcaro: deployments: add list endpoint [repos/cloud/toolforge/components-api] (slavina/add-deploy-delete) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T379093 https://phabricator.wikimedia.org/T379350) (owner: 10sstefanova) [10:53:38] 10Cloud-VPS (Quota-requests): Increase CPUs for webperformancetest - https://phabricator.wikimedia.org/T379514#10309038 (10dcaro) +1 [10:54:58] 10Cloud-VPS (Quota-requests): Increase CPUs for webperformancetest - https://phabricator.wikimedia.org/T379514#10309048 (10fnegri) 05Open→03Resolved ` fnegri@cloudcumin1001:~$ sudo cookbook wmcs.openstack.quota_increase --project webperformancetest --cores 2 --cluster-name eqiad1 --task-id T379514 Loadin... [11:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:09:08] 06cloud-services-team, 10Cloud-VPS, 07Documentation: Improve documentation how to set up a project puppetserver - https://phabricator.wikimedia.org/T379501#10309076 (10fnegri) p:05Triage→03Medium This is caused by a [recent change in Git](https://github.blog/open-source/git/git-security-vulnerability-ann... [11:09:40] 10Tools, 10Pywikibot, 10Pywikibot-Scripts: Implement a webservice at toolforge.org based on create_isbn_edition script - https://phabricator.wikimedia.org/T379488#10309082 (10fnegri) [11:12:02] 06cloud-services-team, 10Cloud-VPS: Members of project-a4b03ac235d443159feaeafec24a92dd not seemingly added to project-bastion - https://phabricator.wikimedia.org/T379478#10309097 (10fnegri) p:05Triage→03High [11:12:14] 06cloud-services-team, 10Cloud-VPS: OpenStack project ids rather than names are being used in hiera settings archiving - https://phabricator.wikimedia.org/T379128#10309100 (10fnegri) p:05Triage→03High [11:12:27] 06cloud-services-team, 10Cloud-VPS, 10Striker: OpenStack project ids rather than names are being used in LDAP group creation - https://phabricator.wikimedia.org/T379030#10309102 (10fnegri) p:05Triage→03High [11:20:39] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Members of project-a4b03ac235d443159feaeafec24a92dd not seemingly added to project-bastion - https://phabricator.wikimedia.org/T379478#10309124 (10fnegri) a:03fnegri [11:22:09] 06cloud-services-team, 10Cloud-VPS, 07LDAP: novaadmin LDAP user is a member of nonexistent LDAP groups - https://phabricator.wikimedia.org/T378847#10309157 (10fnegri) p:05Triage→03Low [11:22:55] 06cloud-services-team, 10Toolforge: mysqldump Command Not Found on tools Bastion - https://phabricator.wikimedia.org/T378882#10309133 (10fnegri) p:05Triage→03Low @LucasWerkmeister I agree we should improve the UX of creating backups (including having streaming logs), but I don't think the right answer is t... [11:33:37] 06cloud-services-team: Temperature clouddumps1001:9290 Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T378844#10309209 (10fnegri) 05Open→03Resolved a:03fnegri This triggered a few times between Nov 1 and Nov 4, then hasn't triggered since. Resolving for now. {F57693671} [11:34:22] 06cloud-services-team, 10Data-Services: Create views for DiscussionTools items tables - https://phabricator.wikimedia.org/T374584#10309223 (10fnegri) p:05Triage→03Medium [11:34:53] 06cloud-services-team, 10Cloud-VPS: Enable use of web proxy for wikiwho.net domain - https://phabricator.wikimedia.org/T376637#10309235 (10fnegri) p:05Triage→03Medium [11:35:55] 06cloud-services-team, 10Cloud-VPS: Migrate projects with floating IPs for web traffic via custom domains to the shared proxy - https://phabricator.wikimedia.org/T378845#10309236 (10fnegri) p:05Triage→03Medium [11:41:50] 06cloud-services-team, 10Citoid: Investigate Wikipedia bot/userscript usage of citoid impacting www.pro-football-reference.com - https://phabricator.wikimedia.org/T378461#10309264 (10fnegri) p:05Triage→03High > * for refill I have just opened a phab ticket: T378686: refill: review citoid usage > * for cita... [11:44:00] 06cloud-services-team, 10Toolforge: Toolforge job fails to find library installed via aptfile - https://phabricator.wikimedia.org/T373251#10309283 (10fnegri) 05Open→03Invalid Resolving as "Invalid" as this is expected behaviour and @bd808 indicated a workaround. [11:44:31] 06cloud-services-team, 10Toolforge: Provide a simple list of the built container images for a given tool (`toolforge build list` subset) - https://phabricator.wikimedia.org/T362836#10309292 (10fnegri) p:05Triage→03Low [11:44:52] 06cloud-services-team, 10Toolforge: [components-api] Add webservice support (to refine) - https://phabricator.wikimedia.org/T362077#10309294 (10fnegri) p:05Triage→03High [11:45:19] 06cloud-services-team, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: kernel message: SGX disabled by BIOS - https://phabricator.wikimedia.org/T379351#10309296 (10fnegri) p:05Triage→03Low [11:45:48] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10309297 (10fnegri) p:05Triage→03Medium [11:45:59] 06cloud-services-team, 10Toolforge: [components-api] Extend the list of build triggers (unrefined) - https://phabricator.wikimedia.org/T362071#10309310 (10fnegri) p:05Triage→03High [11:49:08] 06cloud-services-team, 10Cloud-VPS, 07LDAP: LDAP: review domain and TLS setup - https://phabricator.wikimedia.org/T339909#10309321 (10fnegri) p:05Triage→03Medium [11:52:02] 10Striker, 10Tool-gitlab-account-approval, 10Tool-phab-ban, 10Bitu, and 2 others: Removal of writeapi from siteinfo output breaks all mwclient-based bots, including stashbot (Server Admin Log) - https://phabricator.wikimedia.org/T371977#10309330 (10fnegri) [11:53:53] 06cloud-services-team, 10Toolforge: Allow to remap/rename secrets from envvars when running specific toolforge job - https://phabricator.wikimedia.org/T376849#10309333 (10fnegri) p:05Triage→03Medium [11:54:14] 06cloud-services-team, 10Toolforge, 07Epic: [Hypothesis] WE6.3.6: Toolforge Modernization Discovery - https://phabricator.wikimedia.org/T375914#10309334 (10fnegri) p:05Triage→03High [11:54:46] 06cloud-services-team, 10Data-Services: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#10309336 (10fnegri) p:05Triage→03Medium [11:59:14] FIRING: Kernel warning: Server an-redacteddb1001 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=an-redacteddb1001 - https://alerts.wikimedia.org/?q=alertname%3DKernel+warning [11:59:14] FIRING: Kernel err priority: Server an-redacteddb1001 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=an-redacteddb1001 - https://alerts.wikimedia.org/?q=alertname%3DKernel+err+priority [12:08:23] 06cloud-services-team: openstack: keystone may be failing to add users to the bastion project - https://phabricator.wikimedia.org/T379550 (10aborrero) 03NEW [12:08:53] 06cloud-services-team: openstack: keystone may be failing to add users to the bastion project - https://phabricator.wikimedia.org/T379550#10309381 (10aborrero) p:05Triage→03High [12:08:57] 06cloud-services-team: openstack: keystone may be failing to add users to the bastion project - https://phabricator.wikimedia.org/T379550#10309394 (10aborrero) [12:09:01] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Members of project-a4b03ac235d443159feaeafec24a92dd not seemingly added to project-bastion - https://phabricator.wikimedia.org/T379478#10309395 (10aborrero) [12:12:54] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Members of project-a4b03ac235d443159feaeafec24a92dd not seemingly added to project-bastion - https://phabricator.wikimedia.org/T379478#10309399 (10aborrero) 05Open→03Resolved Fixed. Users should be able to ssh now. Closing this ticket, please reo... [12:19:24] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/22 [12:20:36] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1089730 (owner: 10L10n-bot) [12:31:44] RESOLVED: Kernel warning: Server an-redacteddb1001 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=an-redacteddb1001 - https://alerts.wikimedia.org/?q=alertname%3DKernel+warning [12:31:44] RESOLVED: Kernel err priority: Server an-redacteddb1001 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=an-redacteddb1001 - https://alerts.wikimedia.org/?q=alertname%3DKernel+err+priority [12:39:56] (03approved) 10dcaro: deployments: add list endpoint [repos/cloud/toolforge/components-api] (slavina/add-deploy-delete) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T379093 https://phabricator.wikimedia.org/T379350) (owner: 10sstefanova) [12:39:59] (03update) 10dcaro: deployments: add list endpoint [repos/cloud/toolforge/components-api] (slavina/add-deploy-delete) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T379093 https://phabricator.wikimedia.org/T379350) (owner: 10sstefanova) [12:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:00:27] 10cloud-services-team (FY2024/2025-Q1-Q2), 13Patch-For-Review: Kernel alerts disappear too quickly - https://phabricator.wikimedia.org/T379378#10309484 (10dcaro) A common pattern for this types of checks (ex. counting the amount of errors in a log), is to track the date of each error too, and have a 'timestamp... [13:13:18] (03open) 10dcaro: toolforge_deploy_mr: allow downloading packages from forks [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/211 [13:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:31:07] (03approved) 10dcaro: auth: allow pass through for deploy urls with tokens [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/51 (https://phabricator.wikimedia.org/T362066) [13:31:12] (03merge) 10dcaro: auth: allow pass through for deploy urls with tokens [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/51 (https://phabricator.wikimedia.org/T362066) [13:33:41] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.54-20241111133126-cf520466 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/587 (https://phabricator.wikimedia.org/T362066) [13:33:43] (03open) 10dcaro: functiol: add webservice service.template tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/588 [13:34:00] (03approved) 10dcaro: Warn about unknown keys in service.template [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/59 (owner: 10lucaswerkmeister) [13:36:33] (03update) 10dcaro: functiol: add webservice service.template tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/588 [13:36:55] 10tool-wdlocator: Refresh link for current area - https://phabricator.wikimedia.org/T364287#10309583 (10Samwilson) 05Open→03Resolved a:03Samwilson [13:40:23] 10Cloud-VPS (Quota-requests): Increase CPUs for webperformancetest - https://phabricator.wikimedia.org/T379514#10309586 (10Peter) Thank you! [13:40:31] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [13:41:24] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [13:41:38] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: nova-fullstack: add support for IPv6 - https://phabricator.wikimedia.org/T379356#10309588 (10aborrero) I will use this chance to move the code out of the ops/puppet.git into https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test [13:42:27] (03merge) 10dcaro: Warn about unknown keys in service.template [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/59 (owner: 10lucaswerkmeister) [13:43:30] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [13:43:44] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [13:46:15] 10Cloud-VPS (Quota-requests): Increase RAM and storage for webperformancetest - https://phabricator.wikimedia.org/T379556 (10Peter) 03NEW [13:48:13] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [13:52:41] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate deployment-poolcounter06.deployment-prep.eqiad.wmflabs is about to expire in 16d 23h 58m 30s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [13:52:56] 10cloud-services-team (FY2024/2025-Q1-Q2), 13Patch-For-Review: Kernel alerts disappear too quickly - https://phabricator.wikimedia.org/T379378#10309620 (10fnegri) @dcaro I like this idea. How does it work for the ceph crash entries? Did you write some custom script to check if errors have been "acknowledged"?... [13:53:19] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [13:55:41] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [13:56:06] 06cloud-services-team, 10Citoid: Investigate Wikipedia bot/userscript usage of citoid impacting www.pro-football-reference.com - https://phabricator.wikimedia.org/T378461#10309622 (10Mvolz) >>! In T378461#10309264, @fnegri wrote: >> * for refill I have just opened a phab ticket: T378686: refill: review citoid... [14:01:03] (03update) 10dcaro: functinal: add webservice service.template tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/588 [14:01:11] (03update) 10dcaro: functional: add webservice service.template tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/588 [14:01:57] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [14:02:17] (03approved) 10dcaro: api-gateway: bump to 0.0.54-20241111133126-cf520466 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/587 (https://phabricator.wikimedia.org/T362066) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [14:02:20] (03update) 10dcaro: api-gateway: bump to 0.0.54-20241111133126-cf520466 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/587 (https://phabricator.wikimedia.org/T362066) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [14:02:21] (03merge) 10dcaro: api-gateway: bump to 0.0.54-20241111133126-cf520466 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/587 (https://phabricator.wikimedia.org/T362066) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [14:07:07] 14Toolforge (Toolforge iteration 09): [infra] Add alert when workers have a sustained large amount of D processes - https://phabricator.wikimedia.org/T362093#10309646 (10dcaro) >>! In T362093#10303249, @Leloiandudu wrote: > @dcaro just curious, what happens when the alert goes off? do you manually go and kil... [14:09:25] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics [14:09:28] (03update) 10raymond-ndibe: [toolforge-deploy] kube-state-metrics v5.16.4 --> v5.18.0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/586 (https://phabricator.wikimedia.org/T362867) [14:09:29] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics [14:09:49] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics [14:14:55] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics [14:16:18] (03update) 10dcaro: functional: add webservice service.template tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/588 [14:19:06] 10Tools, 10Pywikibot, 10Pywikibot-Scripts: Implement a webservice at toolforge.org based on create_isbn_edition script - https://phabricator.wikimedia.org/T379488#10309685 (10Geertivp) I have uploaded a new version to GitHub (bug correction, enhanced error messages). https://github.com/geertivp/Pywikibot/blo... [14:21:45] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 05Goal: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206#10309690 (10fnegri) 05Open→03In progress [14:26:21] (03open) 10dcaro: components-api: fix toolsbeta api url [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/589 [14:27:15] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [14:28:16] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 05Goal: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206#10309719 (10fnegri) > Edit: the only other available LTS version is 10.11, so we could consider upgrading to that one instead of 10.6. 11.4 i... [14:33:54] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [14:35:10] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 05Goal: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206#10309757 (10fnegri) My plan is: * create a new replica **tools-db-4** (running Debian Bookworm and MariaDB 10.4) and configure it to replicate... [14:37:40] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' (T352206) [14:37:44] T352206: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206 [14:41:23] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' (T352206) [14:42:48] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud (T352206) [14:42:51] T352206: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206 [14:44:28] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-db-4 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [14:44:40] !log fnegri@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-db-4.tools.eqiad1.wikimedia.cloud (T352206) [14:49:12] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.vm_console [14:49:15] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) [15:02:56] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.openstack.cloudvirt.vm_console [15:03:00] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) [15:03:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:03:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:05:33] (03update) 10raymond-ndibe: [lima-kilo] cache disk for caching container images [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/201 (https://phabricator.wikimedia.org/T378180) [15:05:52] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.openstack.cloudvirt.vm_console [15:05:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:06:03] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [15:06:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:13:44] !log fnegri@cloudcumin1001 deployment-prep START - Cookbook wmcs.vps.refresh_puppet_certs on deployment-poolcounter06.deployment-prep.eqiad1.wikimedia.cloud [15:13:48] !log fnegri@cloudcumin1001 deployment-prep END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on deployment-poolcounter06.deployment-prep.eqiad1.wikimedia.cloud [15:24:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-db-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [15:27:43] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [15:27:48] !log dcaro@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api [15:57:41] RESOLVED: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate deployment-poolcounter06.deployment-prep.eqiad.wmflabs is about to expire in 16d 21h 57m 30s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [15:58:44] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud (T352206) [15:58:48] T352206: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206 [15:59:53] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.openstack.cloudvirt.vm_console [15:59:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:00:05] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [16:00:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:00:24] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.openstack.cloudvirt.vm_console [16:00:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:00:45] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [16:00:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:01:20] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.openstack.cloudvirt.vm_console [16:01:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:01:28] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [16:01:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:01:51] (03update) 10raymond-ndibe: [toolforge-weld] refactor parse_quantity [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/64 (https://phabricator.wikimedia.org/T361120) [16:01:55] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.openstack.cloudvirt.vm_console [16:01:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:02:03] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [16:02:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:02:23] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.openstack.cloudvirt.vm_console [16:02:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:02:30] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [16:02:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:02:48] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.openstack.cloudvirt.vm_console [16:02:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:02:55] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [16:02:55] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-4.tools.eqiad1.wikimedia.cloud (T352206) [16:02:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:03:40] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.openstack.cloudvirt.vm_console [16:03:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:03:48] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [16:03:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:04:28] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-db-4 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [16:04:31] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.openstack.cloudvirt.vm_console [16:04:35] !log dcaro@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [16:06:49] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.openstack.cloudvirt.vm_console [16:06:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:06:57] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [16:06:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:14:28] (03update) 10dcaro: components-api: fix toolsbeta api url [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/589 [16:15:52] (03approved) 10dcaro: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) (owner: 10raymond-ndibe) [16:17:44] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [16:19:24] (03update) 10dcaro: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] (add_cache_disk) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) (owner: 10raymond-ndibe) [16:19:33] (03approved) 10dcaro: [lima-kilo] cache disk for caching container images [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/201 (https://phabricator.wikimedia.org/T378180) (owner: 10raymond-ndibe) [16:19:34] (03update) 10dcaro: [lima-kilo] cache disk for caching container images [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/201 (https://phabricator.wikimedia.org/T378180) (owner: 10raymond-ndibe) [16:20:03] (03update) 10dcaro: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) (owner: 10raymond-ndibe) [16:20:08] (03approved) 10dcaro: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] (add_cache_disk) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) (owner: 10raymond-ndibe) [16:20:58] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tools-db-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:23:09] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [16:30:37] 10cloud-services-team (FY2024/2025-Q1-Q2): [cloudcumin] After the upgrade to spicerack 8.15.2, the wmcs.openstack.cloudvirt.vm_console cookbook stopped working - https://phabricator.wikimedia.org/T379570 (10dcaro) 03NEW [16:30:59] 10cloud-services-team (FY2024/2025-Q1-Q2): [cloudcumin] After the upgrade to spicerack 8.15.2, the wmcs.openstack.cloudvirt.vm_console cookbook stopped working - https://phabricator.wikimedia.org/T379570#10310166 (10dcaro) cc @fnegri [16:31:14] (03update) 10raymond-ndibe: [maintain-harbor] do not clean up images currently running in production [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/35 (https://phabricator.wikimedia.org/T377854) [16:34:07] (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] (allow_getting_config_from_env) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225) [16:39:19] (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] (allow_getting_config_from_env) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225) [16:39:33] (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] (allow_getting_config_from_env) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225) [16:40:32] (03update) 10raymond-ndibe: [toolforge-deploy] deploy maintain-harbor [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/563 (https://phabricator.wikimedia.org/T358225) [16:40:55] (03update) 10raymond-ndibe: [toolforge-deploy] deploy maintain-harbor [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/563 (https://phabricator.wikimedia.org/T358225) [16:49:14] FIRING: Kernel error: Server an-redacteddb1001 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=an-redacteddb1001 - https://alerts.wikimedia.org/?q=alertname%3DKernel+error [16:49:14] FIRING: Kernel warning: Server an-redacteddb1001 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=an-redacteddb1001 - https://alerts.wikimedia.org/?q=alertname%3DKernel+warning [16:49:19] 06cloud-services-team: Kernel error Server an-redacteddb1001 may have kernel errors - https://phabricator.wikimedia.org/T379571 (10phaultfinder) 03NEW [17:03:05] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1089730 (owner: 10L10n-bot) [17:06:15] (03approved) 10dcaro: components-api: fix toolsbeta api url [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/589 [17:06:18] (03merge) 10dcaro: components-api: fix toolsbeta api url [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/589 [17:08:53] 10wikitech.wikimedia.org, 10Education-Program-Dashboard: https://outreachdashboard.wmflabs.org is down - https://phabricator.wikimedia.org/T379572 (10Olea) 03NEW [17:21:53] (03update) 10raymond-ndibe: [maintain-harbor] do not clean up images currently running in production [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/35 (https://phabricator.wikimedia.org/T377854) [17:23:03] (03open) 10dcaro: d/changelog: bump to 0.103.13 [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/60 [17:23:19] (03update) 10raymond-ndibe: [maintain-harbor] do not clean up images currently running in production [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/35 (https://phabricator.wikimedia.org/T377854) [17:23:28] (03update) 10raymond-ndibe: [maintain-harbor] do not clean up images currently running in production [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/35 (https://phabricator.wikimedia.org/T377854) [17:25:17] (03update) 10raymond-ndibe: [maintain-harbor] do not clean up images currently running in production [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/35 (https://phabricator.wikimedia.org/T377854) [17:27:50] (03update) 10raymond-ndibe: [maintain-harbor] do not clean up images currently running in production [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/35 (https://phabricator.wikimedia.org/T377854) [17:28:32] (03update) 10raymond-ndibe: [maintain-harbor] do not clean up images currently running in production [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/35 (https://phabricator.wikimedia.org/T377854) [17:29:56] (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] (allow_getting_config_from_env) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225) [17:34:06] (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] (allow_getting_config_from_env) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225) [17:34:09] (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] (allow_getting_config_from_env) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225) [17:34:29] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [17:34:31] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [17:35:06] (03update) 10raymond-ndibe: [toolforge-deploy] deploy maintain-harbor [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/563 (https://phabricator.wikimedia.org/T358225) [17:53:31] 06cloud-services-team, 10Toolforge, 07Kubernetes: [jobs-api] Allow Toolforge scheduled jobs to have a maximum runtime - https://phabricator.wikimedia.org/T306391#10310315 (10Multichill) @dcaro if you merge a task, can you please copy over relevant information? In this case the fact someone already wrote the... [17:53:48] 06cloud-services-team, 10Toolforge, 07Kubernetes: [jobs-api] Allow Toolforge scheduled jobs to have a maximum runtime - https://phabricator.wikimedia.org/T306391#10310309 (10Multichill) [18:03:04] 06cloud-services-team, 10Toolforge, 07Kubernetes: [jobs-api] Allow Toolforge scheduled jobs to have a maximum runtime - https://phabricator.wikimedia.org/T306391#10310328 (10dcaro) >>! In T306391#10310309, @Multichill wrote: > @dcaro if you merge a task, can you please copy over relevant information? In this... [19:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:45:35] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/22 (owner: 10l10n-bot) [19:45:37] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/22 (owner: 10l10n-bot) [19:48:42] (03open) 10hartman: Fix range error of [A-z] [toolforge-repos/svbot2] - 10https://gitlab.wikimedia.org/toolforge-repos/svbot2/-/merge_requests/1 [19:51:38] (03open) 10hartman: Fix range error in [A-z] [toolforge-repos/logo-test] - 10https://gitlab.wikimedia.org/toolforge-repos/logo-test/-/merge_requests/3 [19:55:37] (03open) 10hartman: Fix error in range of [A-z] [toolforge-repos/eswstool2] - 10https://gitlab.wikimedia.org/toolforge-repos/eswstool2/-/merge_requests/1 [20:03:02] (03open) 10hartman: Fix error in [A-z] range [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/115 [20:06:48] (03open) 10hartman: Fix error in [A-z] range [toolforge-repos/hrwp] - 10https://gitlab.wikimedia.org/toolforge-repos/hrwp/-/merge_requests/2 [20:07:58] (03update) 10hartman: Fix error in [A-z] range [toolforge-repos/hrwp] - 10https://gitlab.wikimedia.org/toolforge-repos/hrwp/-/merge_requests/2 [20:11:02] (03update) 10hartman: Fix range error in [A-z] [toolforge-repos/logo-test] - 10https://gitlab.wikimedia.org/toolforge-repos/logo-test/-/merge_requests/3 [20:26:43] 10wikitech.wikimedia.org, 10MediaWiki-extensions-OATHAuth: Wikitech users being unexpectedly prompted for 2FA tokens - https://phabricator.wikimedia.org/T379398#10310579 (10bd808) 05Open→03Resolved a:03Reedy [20:49:14] FIRING: Kernel error: Server an-redacteddb1001 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=an-redacteddb1001 - https://alerts.wikimedia.org/?q=alertname%3DKernel+error [20:49:14] FIRING: Kernel warning: Server an-redacteddb1001 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=an-redacteddb1001 - https://alerts.wikimedia.org/?q=alertname%3DKernel+warning [21:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:06:48] (03update) 10raymond-ndibe: [maintain-harbor] specify config through environment variables [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/36 (https://phabricator.wikimedia.org/T358225)