[00:15:28] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:20:28] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:17:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:22:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:27:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:40:53] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:05:40] 10Data-Services: enwiki_p database replica has stopped updating - https://phabricator.wikimedia.org/T362732#9725012 (10Liz) Yeah! Life returns to normal. [02:05:56] (SystemdUnitDown) firing: The service unit remove_dangling_cinder_snapshots.service is in failed status on host cloudbackup1001-dev. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1001-dev - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [02:27:37] (03PS1) 10AntiCompositeNumber: StewardBot: add health check script based on canary events [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1021004 [02:28:10] (03CR) 10CI reject: [V:04-1] StewardBot: add health check script based on canary events [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1021004 (owner: 10AntiCompositeNumber) [02:28:14] (03PS2) 10AntiCompositeNumber: StewardBot: add health check script based on canary events [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1021004 [02:28:44] (03CR) 10CI reject: [V:04-1] StewardBot: add health check script based on canary events [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1021004 (owner: 10AntiCompositeNumber) [02:29:47] (03PS3) 10AntiCompositeNumber: StewardBot: add health check script based on canary events [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1021004 [02:38:50] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:58:48] (PuppetConstantChange) firing: (3) Puppet performing a change on every puppet run on cloudcontrol2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [04:00:56] (SystemdUnitDown) firing: The systemd unit remove_dangling_cinder_snapshots.service on node cloudbackup1001-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1001-dev - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [04:01:06] 06cloud-services-team: SystemdUnitDown Unit remove_dangling_cinder_snapshots.service on node cloudbackup1001-dev has been down for long. - https://phabricator.wikimedia.org/T362845 (10phaultfinder) 03NEW [05:45:53] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [06:38:50] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [06:58:48] (PuppetConstantChange) firing: (3) Puppet performing a change on every puppet run on cloudcontrol2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [07:12:45] 10Toolforge, 13Patch-For-Review: Toolforge is missing locales (breaks Mosh and causes various other problems) - https://phabricator.wikimedia.org/T362680#9725188 (10taavi) 05Open→03Resolved a:03taavi [08:00:56] (SystemdUnitDown) firing: The systemd unit remove_dangling_cinder_snapshots.service on node cloudbackup1001-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1001-dev - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:03:15] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Gather feedback from users of the 'unmanaged' debian-12.0-nopuppet image - https://phabricator.wikimedia.org/T355963#9725268 (10fgiunchedi) >>! In T355963#9700818, @Andrew wrote: > >> That's correct yeah, I don't think there are security implications.... [08:41:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:46:41] (CloudVPSDesignateLeaks) firing: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:51:35] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [08:51:41] (CloudVPSDesignateLeaks) firing: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:51:47] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [08:56:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:03:27] (03PS1) 10Majavah: Update @wmde/eslint-config-wikimedia-typescript [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1021379 [09:16:12] 06cloud-services-team, 10Toolforge: [api-gateway] add alert for uptime - https://phabricator.wikimedia.org/T348633#9725556 (10dcaro) [09:16:22] 06cloud-services-team, 10Toolforge: [api-gateway] add alert for uptime - https://phabricator.wikimedia.org/T348633#9725557 (10dcaro) p:05Triage→03High [09:23:47] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [09:23:58] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [09:46:39] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [09:46:52] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [09:50:53] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [10:03:13] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643#9725692 (10dcaro) So far manual tests on the hard drive have been unable to trigger the issue and increase the count... [10:19:21] (03CR) 10Lucas Werkmeister (WMDE): [C:03+1] "Sounds good to me; does that mean we should submit an update here when we release a new version of that librar@?" [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1021379 (owner: 10Majavah) [10:25:58] (03CR) 10Majavah: [C:03+2] "Yes please." [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1021379 (owner: 10Majavah) [10:26:32] (03Merged) 10jenkins-bot: Update @wmde/eslint-config-wikimedia-typescript [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1021379 (owner: 10Majavah) [10:38:50] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [10:58:16] 10Toolforge: Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 (10taavi) 03NEW p:05Triage→03Medium [10:58:20] 10Toolforge: Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#9725847 (10taavi) [10:58:41] 10Toolforge: Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868 (10taavi) 03NEW p:05Triage→03Medium [10:58:47] 10Toolforge: Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868#9725860 (10taavi) [10:58:48] 10Toolforge: Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#9725861 (10taavi) [10:58:48] (PuppetConstantChange) firing: (3) Puppet performing a change on every puppet run on cloudcontrol2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [10:59:34] 10Toolforge: Upgrade Toolforge to Uwubernetes - https://phabricator.wikimedia.org/T362869 (10taavi) 03NEW p:05Triage→03Medium [10:59:40] 10Toolforge: Upgrade Toolforge to Uwubernetes - https://phabricator.wikimedia.org/T362869#9725873 (10taavi) [11:36:25] 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872 (10aborrero) 03NEW [11:40:41] 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9726012 (10dcaro) With the little knowledge of the details, I would lean on enforced validation so far, if anything or anyone is trying to set an option that's not allowed... [11:41:31] 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9726007 (10aborrero) 05Open→03In progress p:05Triage→03Medium [11:51:45] 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9726017 (10aborrero) [11:54:52] 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9726023 (10aborrero) [12:00:56] (SystemdUnitDown) firing: The systemd unit remove_dangling_cinder_snapshots.service on node cloudbackup1001-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1001-dev - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [12:27:34] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1021457 (owner: 10L10n-bot) [12:34:54] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9726134 (10fnegri) [12:34:57] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2007-dev.codfw.wmnet' [12:43:03] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2007-dev.codfw.wmnet' [12:45:47] 10Toolforge: Slow loading on Toolforge - https://phabricator.wikimedia.org/T362822#9726178 (10aborrero) Could you please go to here https://network-tests.toolforge.org/ and download the 1GB file, and report the speed you get? This is the URL: https://network-tests.toolforge.org/files/1GB.bin [12:46:36] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2008-dev.codfw.wmnet' [12:46:50] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudvirt2001-dev.codfw.wmnet' [12:52:31] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2008-dev.codfw.wmnet' [12:53:23] PROBLEM - Check nf_conntrack usage in neutron netns on cloudnet2008-dev is CRITICAL: CRITICAL: no netns defined? https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [12:55:11] ACKNOWLEDGEMENT - Check nf_conntrack usage in neutron netns on cloudnet2008-dev is CRITICAL: CRITICAL: no netns defined? Majavah known - OVS work https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [12:55:36] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet' [12:58:11] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [12:58:20] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) [13:05:00] 10Toolforge: [components-api] add one-off, scheduled and continuous jobs support to the yaml + api - https://phabricator.wikimedia.org/T362075#9726251 (10fnegri) The "wait-for" idea is really nice, but I don't think it's a requirement for "one-off, scheduled and continuous jobs support". Maybe we could start im... [13:13:03] 10Cloud Services Proposals: Decision request - What to use for toolforge components api task execution - https://phabricator.wikimedia.org/T362224#9726290 (10fnegri) I vote for Option 1, I think we should aim for the easiest solution in the first MVP of the component API. I could be wrong, but I don't imagine h... [13:15:19] 10Cloud Services Proposals: Decision request - What to use for toolforge components api task execution - https://phabricator.wikimedia.org/T362224#9726300 (10taavi) Would #1 mean having to keep the single HTTP request alive for the duration of the entire build + deploy? I'd prefer to avoid anything requiring a s... [13:22:47] 10Cloud Services Proposals: Decision request - What to use for toolforge components api task execution - https://phabricator.wikimedia.org/T362224#9726357 (10fnegri) I would also avoid that, but I think Option 1 doesn't necessarily entail a long HTTP request, it could be a trigger+poll mechanism for example. I'm... [13:24:33] (03PS1) 10Slyngshede: Add dummy secrets for idmcloud role. [labs/private] - 10https://gerrit.wikimedia.org/r/1021472 (https://phabricator.wikimedia.org/T362128) [13:27:24] 10Cloud Services Proposals: Decision request - Update python team best practices - https://phabricator.wikimedia.org/T361804#9726371 (10fnegri) I vote B5, with the addition of a link to a "canonical example". I would still write a short summary in the wiki of what is used in the "canonical example", but we could... [13:32:44] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9726419 (10fnegri) I think Option 1 is my preference. > What do we do with the already defined Pod? This is something to verify also in Opt... [13:42:10] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Deploy OVS test setup in codfw1dev - https://phabricator.wikimedia.org/T358761#9726483 (10taavi) The issue above is still persisting on Bobcat. Here's a log of an instance creating where that happened: P60929 [13:45:06] (03PS1) 10Jforrester: releases: Upgrade Codex to 1.4.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1021480 (https://phabricator.wikimedia.org/T362546) [13:45:19] (03CR) 10Jforrester: [C:03+2] releases: Upgrade Codex to 1.4.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1021480 (https://phabricator.wikimedia.org/T362546) (owner: 10Jforrester) [13:45:42] 10Striker: [striker] error when creating tool - https://phabricator.wikimedia.org/T362889 (10dcaro) 03NEW [13:45:53] (03Merged) 10jenkins-bot: releases: Upgrade Codex to 1.4.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1021480 (https://phabricator.wikimedia.org/T362546) (owner: 10Jforrester) [13:47:38] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent - https://phabricator.wikimedia.org/T362233#9726517 (10fnegri) I'm fine with both option 1 and option 3. I don't see a huge practical difference between "we evaluate dropping kyverno" (option 1) or "repl... [13:55:53] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [14:13:10] 10Quarry: [bug] Internal Server Error when trying to Stop Query - https://phabricator.wikimedia.org/T362891 (10Ahecht) 03NEW [14:17:49] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [labs/private] - 10https://gerrit.wikimedia.org/r/1021472 (https://phabricator.wikimedia.org/T362128) (owner: 10Slyngshede) [14:18:12] (03CR) 10Slyngshede: [C:03+2] Add dummy secrets for idmcloud role. [labs/private] - 10https://gerrit.wikimedia.org/r/1021472 (https://phabricator.wikimedia.org/T362128) (owner: 10Slyngshede) [14:18:16] (03CR) 10Slyngshede: [V:03+2 C:03+2] Add dummy secrets for idmcloud role. [labs/private] - 10https://gerrit.wikimedia.org/r/1021472 (https://phabricator.wikimedia.org/T362128) (owner: 10Slyngshede) [14:22:51] (03PS1) 10EoghanGaffney: gitlab: Remove old phabricator key after switching to cfssl [labs/private] - 10https://gerrit.wikimedia.org/r/1021489 [14:25:57] 10Cloud Services Proposals: Decision request - What to use for toolforge components api task execution - https://phabricator.wikimedia.org/T362224#9726698 (10dcaro) >>! In T362224#9726357, @fnegri wrote: > I would also avoid that, but I think Option 1 doesn't necessarily entail a long HTTP request, it could be a... [14:28:01] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [labs/private] - 10https://gerrit.wikimedia.org/r/1021489 (owner: 10EoghanGaffney) [14:33:09] 10Toolforge: [components-api] add one-off, scheduled and continuous jobs support to the yaml + api - https://phabricator.wikimedia.org/T362075#9726729 (10dcaro) >>! In T362075#9726251, @fnegri wrote: > The "wait-for" idea is really nice, but I don't think it's a requirement for "one-off, scheduled and continuous... [14:38:34] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent - https://phabricator.wikimedia.org/T362233#9726744 (10dcaro) >>! In T362233#9726517, @fnegri wrote: > In both cases we are going to try dropping Kyverno after the 1.26 upgrade, unless we find some big b... [14:38:55] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [14:40:28] (PuppetSyncFailure) firing: Failed to update Puppet repository /srv/git/labs/private on instance tools-puppetserver-01 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:42:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:44:05] (03CR) 10Lucas Werkmeister (WMDE): [C:03+1] "Alright, added some instructions at https://github.com/wmde/eslint-config-wikimedia-typescript/pull/49 so we (hopefully) don’t forget." [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1021379 (owner: 10Majavah) [14:45:55] 10Quarry: [bug] Internal Server Error when trying to Stop Query - https://phabricator.wikimedia.org/T362891#9726771 (10SD0001) →14Duplicate dup:03T362213 [14:46:16] 10Quarry: Error 500 when clicking "stop query" - https://phabricator.wikimedia.org/T362213#9726773 (10SD0001) [14:51:08] (03CR) 10Majavah: [C:03+2] "Hmm, this didn't seem to work as easily as I hoped on query-builder.git: https://libraryupgrader2.wmcloud.org/r/wikidata/query-builder?bra" [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1021379 (owner: 10Majavah) [14:52:03] 10Toolforge, 13Patch-For-Review: Toolforge is missing locales (breaks Mosh and causes various other problems) - https://phabricator.wikimedia.org/T362680#9726805 (10LucasWerkmeister) Now it’s working again, thanks a lot! [14:52:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:09:27] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.remove_instance for instance cloud-puppetmaster-05 [15:09:39] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloud-puppetmaster-05 [15:14:02] (03CR) 10VolkerE: [C:03+2] releases: Upgrade Codex to 1.4.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1021480 (https://phabricator.wikimedia.org/T362546) (owner: 10Jforrester) [15:14:28] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on cloudinfra-internal-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [15:23:31] 10Data-Services: Wikimedia Cloud Services Wiki Replicas replication lag in wikidata - https://phabricator.wikimedia.org/T362899#9726912 (10bd808) 05Open→03Declined Current lag is very likely caused by {T352010} as shown on https://wikitech.wikimedia.org/wiki/Map_of_database_maintenance which is linked to... [15:26:31] 10Cloud Services Proposals: Decision request - What to use for toolforge components api task execution - https://phabricator.wikimedia.org/T362224#9726938 (10Slst2020) Option 1 for an MVP, then iterate on it as needed. [15:40:49] (03CR) 10Lucas Werkmeister (WMDE): [C:03+1] "Hm, if I understand correctly, `@typescript-eslint/eslint-plugin` (which is in query-builder’s `devDependencies`) would have to be updated" [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1021379 (owner: 10Majavah) [15:41:58] (03CR) 10Lucas Werkmeister (WMDE): [C:03+1] "Yeah, https://github.com/wmde/new-lexeme-special-page/pull/669/files is the pull request I had in mind – quite a bouquet of packages :/" [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1021379 (owner: 10Majavah) [15:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:45:39] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent - https://phabricator.wikimedia.org/T362233#9727034 (10Slst2020) Just acknowledging that I've seen this discussion, but that I don't know enough about this topic to have a preference. [15:47:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:52:50] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Deploy OVS test setup in codfw1dev - https://phabricator.wikimedia.org/T358761#9727060 (10taavi) The firewall issue was fixed by the above patch setting the firewall driver to the same value on all config files. I can now create an i... [16:00:56] (SystemdUnitDown) firing: The systemd unit remove_dangling_cinder_snapshots.service on node cloudbackup1001-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1001-dev - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:03:57] (03CR) 10David Caro: "Yep, not proud of it either" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/990976 (owner: 10David Caro) [16:04:06] (03PS7) 10David Caro: ceph.drain_osd_node: improve logs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/990976 [16:32:16] 10Cloud Services Proposals: Decision request - Update python team best practices - https://phabricator.wikimedia.org/T361804#9727168 (10dcaro) 05Open→03Resolved a:03dcaro Unless I understood wrongly, we seem to agree on going with option **B5** with an example repository, I'll do that and close this de... [16:37:09] 10Cloud Services Proposals: Decision request - Update python team best practices - https://phabricator.wikimedia.org/T361804#9727240 (10dcaro) [16:42:40] 10Striker: [striker] error when creating tool - https://phabricator.wikimedia.org/T362889#9727272 (10bd808) The tool in question is https://toolsadmin.wikimedia.org/tools/id/api-docs. Things are definitely goofy here with there being no toolinfo record. There are several related log events in logstash, and they... [16:50:34] 10Striker: Archive Phabricator project tags for disabled tools - https://phabricator.wikimedia.org/T362313#9727330 (10bd808) This is basically a missing integration with the system that @andrew built for {T170355} ([[https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Deleting_a_tool|wikitech docs]]). The... [16:54:57] 10Cloud Services Proposals: Decision request - Update python team best practices - https://phabricator.wikimedia.org/T361804#9727360 (10dcaro) I wrote https://wikitech.wikimedia.org/wiki/User:David_Caro/Coding_proposal as a generic coding page, as the pre-commit checks and such include golang, shell, yaml an... [16:54:57] 10Cloud Services Proposals: Decision request - What to use for toolforge components api task execution - https://phabricator.wikimedia.org/T362224#9727358 (10fnegri) > I would mean one long HTTP request yes, that's what I meant with synchronous. I see, thanks for clarifying. If we take the simple case of a too... [16:57:27] 10Cloud Services Proposals: Decision request - Update python team best practices - https://phabricator.wikimedia.org/T361804#9727383 (10fnegri) Sounds good to me, thanks! [17:03:43] 10Striker, 10Phabricator: Diffusion mirrors of GitLab repos failing to be created by Striker - https://phabricator.wikimedia.org/T362909 (10bd808) 03NEW [17:04:47] 10Cloud Services Proposals: Decision request - What to use for toolforge components api task execution - https://phabricator.wikimedia.org/T362224#9727408 (10dcaro) >>! In T362224#9727358, @fnegri wrote: >> I would mean one long HTTP request yes, that's what I meant with synchronous. > > I see, thanks for clari... [17:11:44] 10Toolforge: Slow loading on Toolforge - https://phabricator.wikimedia.org/T362822#9727421 (10GPSLeo) The loading speed is currently much faster than yesterday. I especially experienced the problem with the vector tile loading on my tool https://uploadmap.toolforge.org/protectedareaseu.html. Here are some speed... [17:12:31] 10Striker, 10Phabricator: Diffusion mirrors of GitLab repos failing to be created by Striker - https://phabricator.wikimedia.org/T362909#9727422 (10bd808) I was able to create {R3385} via the Phabricator UI. Is it only the Conduit API that is failing? Are the failures intermittent? [17:30:28] (PuppetSyncFailure) resolved: Failed to update Puppet repository /srv/git/labs/private on instance tools-puppetserver-01 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [17:33:01] 10Striker, 10Phabricator: Diffusion mirrors of GitLab repos failing to be created by Striker - https://phabricator.wikimedia.org/T362909#9727521 (10bd808) [17:33:03] 10Striker: Mirror Striker managed GitLab repos in Diffusion - https://phabricator.wikimedia.org/T317345#9727522 (10bd808) [17:35:24] 10Striker: Tool created following initial name validation failure is missing toolinfo record and having other problems with GitLab repositories - https://phabricator.wikimedia.org/T362889#9727540 (10bd808) [17:52:41] (CloudVPSDesignateLeaks) firing: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:57:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:00:33] 10Striker, 10Phabricator: Diffusion mirrors of GitLab repos failing to be created by Striker - https://phabricator.wikimedia.org/T362909#9727661 (10Aklapper) @bd808: Does this error still appear after 2024-04-16 15:30UTC? If not, then this is likely a duplicate of T355644 (which reverted T352530 which I ideall... [18:00:53] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [18:25:15] 10Tool-extjsonuploader, 10GitLab (Integrations): Support gitlab.wikimedia.org in extjsonuploader - https://phabricator.wikimedia.org/T352831#9727782 (10brennen) [18:27:46] 10Quarry, 10Toolforge, 10ChangeProp, 06collaboration-services, and 10 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9727771 (10brennen) [18:38:51] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [18:52:18] 10Cloud Services Proposals: Decision request - Update python team best practices - https://phabricator.wikimedia.org/T361804#9727897 (10Andrew) I'm late to this, but I also agree with B5. Do we need another rule regarding what to do with existing code when applying new checks or standards and would result in... [19:06:34] (DiskSpace) firing: Disk space cloudbackup1002-dev:9100:/srv/cinder-backups 1.203% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:15:34] PROBLEM - Disk space on cloudbackup1002-dev is CRITICAL: DISK CRITICAL - free space: /srv/cinder-backups 228MiB (1% inode=99%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbackup1002-dev&var-datasource=eqiad+prometheus/ops [19:34:39] 10Striker, 10Phabricator: Diffusion mirrors of GitLab repos failing to be created by Striker - https://phabricator.wikimedia.org/T362909#9728028 (10bd808) >>! In T362909#9727661, @Aklapper wrote: > @bd808: Does this error still appear after 2024-04-16 15:30UTC? If not, then this is likely a duplicate of T35564... [19:41:41] (CloudVPSDesignateLeaks) firing: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:46:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:51:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:55:34] RECOVERY - Disk space on cloudbackup1002-dev is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbackup1002-dev&var-datasource=eqiad+prometheus/ops [19:56:34] (DiskSpace) resolved: Disk space cloudbackup1002-dev:9100:/srv/cinder-backups 1.203% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:56:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:00:56] (SystemdUnitDown) firing: The systemd unit remove_dangling_cinder_snapshots.service on node cloudbackup1001-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1001-dev - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [20:15:46] (03CR) 10Dzahn: "not really gitlab-related but lgtm :)" [labs/private] - 10https://gerrit.wikimedia.org/r/1021489 (owner: 10EoghanGaffney) [20:16:25] (03PS2) 10Dzahn: ssl: Remove old phabricator key after switching to cfssl [labs/private] - 10https://gerrit.wikimedia.org/r/1021489 (owner: 10EoghanGaffney) [20:16:39] (03CR) 10Dzahn: [C:03+1] ssl: Remove old phabricator key after switching to cfssl [labs/private] - 10https://gerrit.wikimedia.org/r/1021489 (owner: 10EoghanGaffney) [20:25:45] (03CR) 10EoghanGaffney: [C:03+2] ssl: Remove old phabricator key after switching to cfssl [labs/private] - 10https://gerrit.wikimedia.org/r/1021489 (owner: 10EoghanGaffney) [20:25:49] (03CR) 10EoghanGaffney: [V:03+2 C:03+2] ssl: Remove old phabricator key after switching to cfssl [labs/private] - 10https://gerrit.wikimedia.org/r/1021489 (owner: 10EoghanGaffney) [20:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:47:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:52:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:52:56] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:57:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:05:53] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [22:36:55] 10Wikibugs, 10GitLab (Integrations): Automate setup of comment, pipeline, and job webhooks for all GitLab projects - https://phabricator.wikimedia.org/T362940 (10bd808) 03NEW [22:37:43] 10Wikibugs, 10GitLab (Integrations): Automate setup of comment, pipeline, and job webhooks for all GitLab projects - https://phabricator.wikimedia.org/T362940#9728377 (10bd808) p:05Triage→03Medium [22:38:51] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [23:02:57] 10Wikibugs, 10GitLab (Integrations): Automate setup of comment, pipeline, and job webhooks for all GitLab projects - https://phabricator.wikimedia.org/T362940#9728392 (10brennen) I guess my other thought about a home for this is that it could live in [[ https://gitlab.wikimedia.org/repos/releng/gitlab-settings... [23:09:16] (03CR) 10Dzahn: [V:03+2 C:03+2] delete graphite.discovery.wmnet dummy key [labs/private] - 10https://gerrit.wikimedia.org/r/1019889 (https://phabricator.wikimedia.org/T360414) (owner: 10Dzahn) [23:09:21] (03PS2) 10Dzahn: delete graphite.discovery.wmnet dummy key [labs/private] - 10https://gerrit.wikimedia.org/r/1019889 (https://phabricator.wikimedia.org/T360414) [23:11:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:12:55] (03CR) 10Dzahn: [V:03+2 C:03+2] delete graphite.discovery.wmnet dummy key [labs/private] - 10https://gerrit.wikimedia.org/r/1019889 (https://phabricator.wikimedia.org/T360414) (owner: 10Dzahn) [23:16:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:26:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks