[00:35:01] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196#12027456 (10komla) [00:42:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [00:47:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [01:15:49] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.504% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [01:33:05] 10Tool-centralnotice-banner-editor: Change the templates to dark mode accepted colours as a default - https://phabricator.wikimedia.org/T429069#12027477 (10Oyelola_Victoria) Hi @Ciell, thanks for flagging this. The templates now default to tokens and no longer raise the custom colour warning. {F89009661} [02:35:34] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.717% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:54:27] (03open) 10shadabgdg: Rename test URL query parameter to avoid conflictsRename testing URL query parameter [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/18 (https://phabricator.wikimedia.org/T421542) [03:58:36] 06cloud-services-team, 10Toolforge, 13Patch-For-Review, 07patch-welcome: Toolforge 404 handler displays wrong tool name from ?url= query parameter - https://phabricator.wikimedia.org/T421542#12027560 (10Shadabgdg) Submitted a merge request implementing the proposed renaming of the testing query parameter f... [05:31:12] 06cloud-services-team, 10Cloud-VPS: Openstack backups postgres cursor log errors - https://phabricator.wikimedia.org/T428865#12027622 (10Volans) p:05High→03Medium [05:47:48] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Ditch wmcs-specific alerts for systemdunitdown in favor of production alerts - https://phabricator.wikimedia.org/T428873#12027639 (10Volans) a:05Volans→03None I think it was assigned to me because of a child task of th... [05:51:03] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Toolforge, 13Patch-For-Review: Add tracing to understand Toolforge and CloudVPS usage and dependencies - https://phabricator.wikimedia.org/T399313#12027644 (10Volans) p:05High→03Low a:05Volans→03None For now we will stay with the NFS tracing... [06:03:00] 06cloud-services-team, 10Data-Services, 10VPS-Projects: Requesting access to NFS mount /public/dumps for language Cloud VPS project - https://phabricator.wikimedia.org/T429433 (10santhosh) 03NEW [07:34:08] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196#12027872 (10komla) The message has gone out to the admins [07:35:14] 10Cloud-Services, 06Moderator-Tools-Team, 07Epic: Migrate Moderator Tools Cloud VPS tools from Debian Bullseye - https://phabricator.wikimedia.org/T429442#12027881 (10Samwalton9-WMF) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/... [07:37:19] 10Cloud-Services, 06Moderator-Tools-Team, 07Epic: Migrate Moderator Tools Cloud VPS tools from Debian Bullseye - https://phabricator.wikimedia.org/T429442#12027888 (10Samwalton9-WMF) [07:37:21] 10Cloud-Services, 06Moderator-Tools-Team, 07Epic: Migrate Moderator Tools Cloud VPS tools from Debian Bullseye - https://phabricator.wikimedia.org/T429442#12027891 (10Samwalton9-WMF) p:05Triage→03High [07:37:38] 10Cloud-VPS (Debian Bullseye Deprecation), 06Moderator-Tools-Team, 07Epic: Migrate Moderator Tools Cloud VPS tools from Debian Bullseye - https://phabricator.wikimedia.org/T429442#12027892 (10Samwalton9-WMF) [07:38:40] 10Cloud-VPS (Debian Bullseye Deprecation), 06Moderator-Tools-Team, 06The-Wikipedia-Library: wikilink: Replace deprecated Bullseye VM in Cloud VPS - https://phabricator.wikimedia.org/T402055#12027909 (10Samwalton9-WMF) [07:39:02] 10Cloud-VPS (Debian Bullseye Deprecation), 06Moderator-Tools-Team, 06The-Wikipedia-Library: wikilink: Replace deprecated Bullseye VM in Cloud VPS - https://phabricator.wikimedia.org/T402055#12027911 (10Samwalton9-WMF) [07:39:18] 10Cloud-VPS (Debian Bullseye Deprecation), 06Moderator-Tools-Team, 06The-Wikipedia-Library: twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12027913 (10Samwalton9-WMF) [07:39:28] 10Cloud-VPS (Debian Bullseye Deprecation), 06Moderator-Tools-Team, 06The-Wikipedia-Library: twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12027917 (10Samwalton9-WMF) [07:40:07] 10Cloud-VPS (Debian Bullseye Deprecation), 06Moderator-Tools-Team, 06The-Wikipedia-Library: twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12027920 (10Samwalton9-WMF) p:05Triage→03High [07:40:24] 10Cloud-VPS (Debian Bullseye Deprecation), 06Moderator-Tools-Team, 07Epic: Migrate Moderator Tools Cloud VPS tools from Debian Bullseye - https://phabricator.wikimedia.org/T429442#12027921 (10Samwalton9-WMF) [07:40:27] 10Cloud-VPS (Debian Bullseye Deprecation), 06Moderator-Tools-Team, 06The-Wikipedia-Library: wikilink: Replace deprecated Bullseye VM in Cloud VPS - https://phabricator.wikimedia.org/T402055#12027924 (10Samwalton9-WMF) [07:40:31] 10Cloud-VPS (Debian Bullseye Deprecation), 06Moderator-Tools-Team, 06The-Wikipedia-Library: twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12027925 (10Samwalton9-WMF) [07:42:27] (03approved) 10filippo: Release version 0.2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/13 (https://phabricator.wikimedia.org/T351637) (owner: 10fnegri) [07:44:34] 06cloud-services-team, 10Horizon: IDP/SSO logout behavior is weird - https://phabricator.wikimedia.org/T374123#12027932 (10Samwalton9-WMF) I just ran into this frustration today - I logged in to the wrong account and then couldn't log out of Horizon because it just kept logging me back in. Wasn't sure how to r... [07:50:05] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Ditch wmcs-specific alerts for systemdunitdown in favor of production alerts - https://phabricator.wikimedia.org/T428873#12027946 (10fgiunchedi) a:03fgiunchedi that's correct ! [07:52:57] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Replace https://os-deprecation.toolforge.org/ with something that handles in-place upgraded hosts - https://phabricator.wikimedia.org/T428919#12027956 (10fgiunchedi) I agree -- fresh build is best, and in-place upgrade is also acceptable [07:59:56] 06cloud-services-team, 10Horizon: IDP/SSO logout behavior is weird - https://phabricator.wikimedia.org/T374123#12027984 (10Volans) p:05Low→03Medium And from a quick test logging out of IDP doesn't actually log you out, repro steps: * Open incognito window * Go to horizon, login via IDP * Go to https://idp.... [08:00:09] 06cloud-services-team, 10Data-Services, 10VPS-Projects: Requesting access to NFS mount /public/dumps for language Cloud VPS project - https://phabricator.wikimedia.org/T429433#12027990 (10dcaro) +1 from me [08:00:19] 06cloud-services-team, 10Horizon: Horizon IDP/SSO logout doesn't really logout - https://phabricator.wikimedia.org/T374123#12027992 (10Volans) [08:00:29] 06cloud-services-team, 10Cloud-VPS: Add/review linting of flavors in opentofu - https://phabricator.wikimedia.org/T429336#12027993 (10fgiunchedi) I'm thinking we can either checking the tofu syntax, or probably more realistically check the resulting state at CI time for our constraints. That would open the doo... [08:10:44] 06cloud-services-team: cloudceph "HEALTH_WARN 17 OSD(s) experiencing slow operations in BlueStore" - https://phabricator.wikimedia.org/T429387#12028019 (10fgiunchedi) Later in the thread there's a few suggested troubleshooting steps, how do those look? https://www.mail-archive.com/ceph-users@ceph.io/msg30032.html [08:14:21] 10Cloud-VPS (Project-requests): Deletion of project maps - https://phabricator.wikimedia.org/T429446 (10Volans) 03NEW [08:15:36] (03update) 10dcaro: core.images.from_short_name_or_url: add aliases for unknown image [repos/cloud/toolforge/jobs-api] (fix_harbor_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/311 [08:16:02] 10Cloud-VPS (Project-requests): Deletion of project netops-clab - https://phabricator.wikimedia.org/T429447 (10Volans) 03NEW [08:17:15] 10Tool-centralnotice-banner-editor: Change the templates to dark mode accepted colours as a default - https://phabricator.wikimedia.org/T429069#12028049 (10Oyelola_Victoria) 05Open→03Resolved [08:17:18] 10Cloud-VPS (Project-requests): Deletion of project maps - https://phabricator.wikimedia.org/T429446#12028050 (10elukey) +1 [08:21:08] 10Cloud-VPS (Project-requests): Deletion of project maps - https://phabricator.wikimedia.org/T429446#12028056 (10MoritzMuehlenhoff) Can be removed [08:28:46] (03approved) 10dcaro: Release version 0.2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/13 (https://phabricator.wikimedia.org/T351637) (owner: 10fnegri) [08:29:49] 10Cloud-VPS (Project-requests): Deletion of project netops-clab - https://phabricator.wikimedia.org/T429447#12028078 (10cmooney) Thanks @volans, I probably should have done this myself a while ago. But yes I think it can go. I’ve made an image available to other staff so they can run themselves if needed, I do... [08:31:59] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review, 07patch-welcome: Toolforge 404 handler displays wrong tool name from ?url= query parameter - https://phabricator.wikimedia.org/T421542#12028084 (10dcaro) [08:46:28] 10Cloud-VPS (Project-requests): Deletion of project maps - https://phabricator.wikimedia.org/T429446#12028163 (10Jgiannelos) Yes, it can be removed. [08:53:21] (03update) 10fnegri: envvars-admission: bump to 0.0.41-20260616161814-962b885d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1301 (https://phabricator.wikimedia.org/T321316 https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:53:40] (03merge) 10fnegri: envvars-admission: bump to 0.0.41-20260616161814-962b885d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1301 (https://phabricator.wikimedia.org/T321316 https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:55:24] (03update) 10fnegri: core.images._get_harbor_images: fix wrong stale cache check [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/310 (owner: 10dcaro) [08:55:25] (03update) 10fnegri: core.images._get_harbor_images: fix wrong stale cache check [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/310 (owner: 10dcaro) [08:55:30] (03approved) 10fnegri: core.images._get_harbor_images: fix wrong stale cache check [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/310 (owner: 10dcaro) [08:56:22] (03update) 10fnegri: core.images.from_short_name_or_url: add aliases for unknown image [repos/cloud/toolforge/jobs-api] (fix_harbor_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/311 (owner: 10dcaro) [08:56:23] (03update) 10fnegri: core.images.from_short_name_or_url: add aliases for unknown image [repos/cloud/toolforge/jobs-api] (fix_harbor_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/311 (owner: 10dcaro) [08:56:28] (03approved) 10fnegri: core.images.from_short_name_or_url: add aliases for unknown image [repos/cloud/toolforge/jobs-api] (fix_harbor_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/311 (owner: 10dcaro) [09:04:47] (03update) 10dcaro: _update_storage_job_status_from_runtime: exclude image.exists/status [repos/cloud/toolforge/jobs-api] (fix_aliases) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/312 [09:05:01] (03update) 10fnegri: Release version 0.2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/13 (https://phabricator.wikimedia.org/T351637) [09:05:30] (03update) 10fnegri: Release version 0.2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/13 (https://phabricator.wikimedia.org/T351637) [09:10:13] (03update) 10fnegri: Release version 0.2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/13 (https://phabricator.wikimedia.org/T351637) [09:11:34] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team: Establish a blackbox network probe vantage point into cloud realm - https://phabricator.wikimedia.org/T429451 (10fgiunchedi) 03NEW [09:12:16] 06cloud-services-team, 10Toolforge, 07Documentation, 07good first task: Create a new doc about managing and sharing files in Toolforge - https://phabricator.wikimedia.org/T347753#12028279 (10Tejinderk.2004) Heyy! I'm currently working on this task to consolidate Toolforge file management documentation int... [09:12:54] (03update) 10fnegri: Release version 0.2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/13 (https://phabricator.wikimedia.org/T351637) [09:15:23] (03merge) 10fnegri: Release version 0.2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/13 (https://phabricator.wikimedia.org/T351637) [09:17:14] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Establish a blackbox network probe vantage point into cloud realm - https://phabricator.wikimedia.org/T429451#12028306 (10fgiunchedi) cc #netops too for their input on the idea [09:18:36] (03update) 10dcaro: _update_storage_job_status_from_runtime: exclude image.exists/status [repos/cloud/toolforge/jobs-api] (fix_aliases) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/312 [09:18:56] (03update) 10dcaro: _update_storage_job_status_from_runtime: exclude image.exists/status [repos/cloud/toolforge/jobs-api] (fix_aliases) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/312 [09:38:02] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196#12028389 (10valerio.bozzolan) Thanks! Received. Discussing a plan right now for `glams` {T416697} [09:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:38:52] 10Cloud-VPS (Project-requests): Deletion of project maps - https://phabricator.wikimedia.org/T429446#12028399 (10MoritzMuehlenhoff) >>! In T429446#12028163, @Jgiannelos wrote: > Its not used by CTT team so we don't need it. Can somebody check with the owners though ? Which owners should that be? [09:41:34] 10Cloud-VPS (Project-requests): Deletion of project maps - https://phabricator.wikimedia.org/T429446#12028424 (10Jgiannelos) From the deprecation report I see @Chippyy and @Andrew [09:59:19] (03update) 10fnegri: deploy_task: wait for empty slot when starting build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/173 (owner: 10dcaro) [10:01:24] (03update) 10fnegri: _update_storage_job_status_from_runtime: exclude image.exists/status [repos/cloud/toolforge/jobs-api] (fix_aliases) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/312 (owner: 10dcaro) [10:50:33] 10Cloud-VPS (Project-requests): Deletion of project maps - https://phabricator.wikimedia.org/T429446#12028614 (10Chippyy) I can speak for maps-warper4 instance on maps, it's actively being used for the Wikimaps Warper Commons Project: https://warper.wmflabs.org/ Other instances are also being used. E.g. WikiMi... [11:02:32] 10Cloud-VPS (Project-requests): Deletion of project netops-clab - https://phabricator.wikimedia.org/T429447#12028674 (10Volans) Thanks for clearing up the resources, I'll take care of the rest. [11:02:32] !log volans@cloudcumin1001 netops-clab START - Cookbook wmcs.vps.delete_project for project netops-clab in eqiad1 (T429447) [11:02:36] T429447: Deletion of project netops-clab - https://phabricator.wikimedia.org/T429447 [11:03:07] (03update) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project netops-clab [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/318 (https://phabricator.wikimedia.org/T429447) [11:03:09] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project netops-clab [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/318 (https://phabricator.wikimedia.org/T429447) [11:06:20] volans@cloudcumin1001 delete_project (PID 3656018) is awaiting input [11:27:17] 10Toolforge, 06tools-platform-team: [logs-api] Using --latest N gives the first N lines since the last hour, not the last - https://phabricator.wikimedia.org/T419863#12028781 (10aputhin) [11:58:17] 10Tools, 06Commons, 07Community-Wishlist: Image Editor for Commons - https://phabricator.wikimedia.org/T426106#12028900 (10mikez-WMF) Ah thank you @Doc_James - I didn't know about this! [12:14:50] (03approved) 10dcaro: projects: delete project netops-clab [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/318 (https://phabricator.wikimedia.org/T429447) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [12:29:30] (03merge) 10volans: projects: delete project netops-clab [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/318 (https://phabricator.wikimedia.org/T429447) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [12:34:55] !log volans@cloudcumin1001 netops-clab END (PASS) - Cookbook wmcs.vps.delete_project (exit_code=0) for project netops-clab in eqiad1 (T429447) [12:34:59] T429447: Deletion of project netops-clab - https://phabricator.wikimedia.org/T429447 [12:36:04] 10Cloud-VPS (Project-requests): Deletion of project netops-clab - https://phabricator.wikimedia.org/T429447#12028996 (10Volans) 05Open→03Resolved a:03Volans The project is no more, thanks for freeing up resources. [12:37:51] 10Toolforge, 06tools-platform-team: [image-config] deprecate and move all data to builds-api - https://phabricator.wikimedia.org/T409728#12029007 (10aputhin) [12:38:03] 06cloud-services-team, 10Toolforge, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: [build-service] user-story 14: Run a set of security checks on the full service - https://phabricator.wikimedia.org/T325208#12029008 (10dcaro) 05Open→03Declined Not relevant anymore, will create a n... [12:38:41] 06cloud-services-team, 10Toolforge, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 07Epic: [builds-cli,builds-api,builds-builder,jobs-cli,josb-api] user-story 14: I want have some certainty that the service is secure - https://phabricator.wikimedia.org/T325207#12029012 (10dcaro) 05... [12:39:13] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [components-api] optionally log deployments to SAL automatically - https://phabricator.wikimedia.org/T393169#12029015 (10dcaro) [12:40:07] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [components-api] add order to the components deployment - https://phabricator.wikimedia.org/T362075#12029017 (10dcaro) [12:41:37] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [webservice,toolforge-cli] Make `webservice shell` a standalone tool - https://phabricator.wikimedia.org/T311917#12029019 (10dcaro) [12:44:52] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [components-api] Add source polling build trigger - https://phabricator.wikimedia.org/T362071#12029027 (10dcaro) [12:45:48] 06cloud-services-team, 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-api] Refactor before webservice support - https://phabricator.wikimedia.org/T359804#12029030 (10dcaro) [12:47:07] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [jobs-api] separate jobs-framework k8s object templates from code - https://phabricator.wikimedia.org/T358815#12029032 (10dcaro) [12:47:32] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [jobs-api] Generate the openapi definition from the code - https://phabricator.wikimedia.org/T390138#12029034 (10dcaro) [12:48:12] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [components-api,beta] Generated configs should contain cpu values as numbers, not strings - https://phabricator.wikimedia.org/T398497#12029036 (10dcaro) [12:58:14] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: [toolforge-cli.build] Implement a --json flag to output machine-readable output - https://phabricator.wikimedia.org/T334589#12029057 (10dcaro) [12:59:19] 06cloud-services-team, 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] optionally log deployments to SAL automatically - https://phabricator.wikimedia.org/T393169#12029068 (10dcaro) [13:12:57] 10Cloud-VPS (Project-requests), 10Maps: Deletion of project maps - https://phabricator.wikimedia.org/T429446#12029124 (10Volans) @elukey , @MoritzMuehlenhoff , I think this might have been a bit rushed, I see activity there... and the list of admins in Horizon is long: {F89112472} Adding more people/tags for... [13:13:52] 10Cloud-VPS (Project-requests), 10Maps: Deletion of project maps - https://phabricator.wikimedia.org/T429446#12029139 (10Volans) Sorry I was writing my reply while gathering the info and missed the previous replies... [13:15:07] 10Cloud-VPS (Project-requests), 10Maps: Check current usage of project maps on CloudVPS - https://phabricator.wikimedia.org/T429446#12029142 (10Volans) [13:38:41] FIRING: CloudVPSDesignateLeaks: Detected 10 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 10 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:52:56] (03update) 10dcaro: _update_storage_job_status_from_runtime: exclude image.exists/status [repos/cloud/toolforge/jobs-api] (fix_aliases) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/312 [14:16:42] 10Toolforge, 06tools-platform-team: [jobs-api] separate jobs-framework k8s object templates from code - https://phabricator.wikimedia.org/T358815#12029591 (10aputhin) p:05High→03Low [14:17:00] 10Toolforge, 06tools-platform-team: [jobs-api] separate jobs-framework k8s object templates from code - https://phabricator.wikimedia.org/T358815#12029596 (10aputhin) Needs more information regarding priority - what's the expected impact? [14:18:02] 10Toolforge, 06tools-platform-team: [jobs-api] Generate the openapi definition from the code - https://phabricator.wikimedia.org/T390138#12029611 (10aputhin) [14:26:18] 06cloud-services-team, 10wikitech.wikimedia.org, 10Observability-Alerting: Move wikitech-static monitoring off Icinga - https://phabricator.wikimedia.org/T362397#12029640 (10fgiunchedi) I ran a quick audit on the wikitech-static icinga monitoring today: * wikitech-static.wikimedia.org no longer answers ping... [14:34:09] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] allow specifying `source_repo`+`ref` for the config - https://phabricator.wikimedia.org/T402764#12029696 (10aputhin) [14:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:47:13] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering, 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497 (10Ottomata) 03NEW [14:48:32] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering, 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12029785 (10Ottomata) cc #editing_qa and #test_kitchen: Do you all use event platform in be... [14:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:50:55] 10Tool-wikimedia-attribution, 10MediaWiki-REST-API, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Check Editor Counts - https://phabricator.wikimedia.org/T427548#12029792 (10HCoplin-WMF) a:05HCoplin-WMF→03pmiazga @pmiazga will take a look at this this week. The recommendation from our sync was... [14:53:30] 10Toolforge, 06tools-platform-team: [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#12029796 (10aputhin) [14:58:19] 10Tool-wikimedia-attribution, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Check Editor Counts - https://phabricator.wikimedia.org/T427548#12029812 (10HCoplin-WMF) [15:02:11] 06cloud-services-team, 10wikitech.wikimedia.org, 10Observability-Alerting: Move wikitech-static monitoring off Icinga - https://phabricator.wikimedia.org/T362397#12029825 (10fgiunchedi) Nowadays the "content in sync" signal can also be read as "how long ago did the docker container CI pipeline last succeed?"... [15:28:53] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 07Epic: [KR] WE6.3 Introduce a sustainability scoring system for the Toolforge platform - https://phabricator.wikimedia.org/T368600#12029979 (10dcaro) 05In progress→03Resolved Closing this, as this is to keep track of a KR than already passed, @... [15:29:29] 10Toolforge, 06tools-platform-team: [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#12029997 (10dcaro) [15:29:30] 14cloud-services-team (FY2024/2025-Q1-Q2), 14Toolforge (Toolforge iteration 16), 13Patch-For-Review: [components-api] Get a skeleton of API webservice and implement `/tool//deploy` with single continuous job deployment only - https://phabricator.wikimedia.org/T362069#12029998 (10dcaro) [15:37:55] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#12030023 (10dcaro) [15:38:29] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-api] support exposing continuous jobs to the internet - https://phabricator.wikimedia.org/T423408#12030033 (10dcaro) [15:38:32] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#12030035 (10dcaro) [15:38:33] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: allow exposing continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#12030034 (10dcaro) [15:38:43] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-cli] support publishing continuous job to the internet - https://phabricator.wikimedia.org/T423410#12030040 (10dcaro) [15:38:50] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: allow exposing continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#12030041 (10dcaro) [15:38:53] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#12030042 (10dcaro) [15:40:26] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: allow exposing continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#12030051 (10dcaro) →14Duplicate dup:03T348755 [15:40:27] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#12030053 (10dcaro) [15:55:47] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-cli, jobs-api] support --webservice option - https://phabricator.wikimedia.org/T428898#12030115 (10dcaro) [15:56:31] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-cli, jobs-api] support --webservice option - https://phabricator.wikimedia.org/T428898#12030121 (10dcaro) [15:57:50] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06tools-platform-team, 13Patch-For-Review: [wikireplicas] add proper dry-run/diff mode to maintain-views - https://phabricator.wikimedia.org/T351637#12030126 (10fnegri) The new version of `maintain-views` is now deployed to all clouddb* hosts. [15:58:36] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-cli] support publishing continuous job to the internet - https://phabricator.wikimedia.org/T423410#12030136 (10dcaro) [15:58:37] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-api] support exposing continuous jobs to the internet - https://phabricator.wikimedia.org/T423408#12030135 (10dcaro) [15:58:44] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#12030137 (10dcaro) [16:00:14] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [webservice,toolforge-cli] Make `webservice shell` a standalone tool - https://phabricator.wikimedia.org/T311917#12030159 (10dcaro) [16:00:19] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#12030160 (10dcaro) [16:00:27] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [webservice,toolforge-cli] Make `webservice shell` a standalone tool - https://phabricator.wikimedia.org/T311917#12030161 (10dcaro) [16:00:33] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#12030162 (10dcaro) [16:06:58] 10Toolforge, 06tools-platform-team: [lima-kilo] increase the number of inotify watches - https://phabricator.wikimedia.org/T429347#12030194 (10dcaro) 05Open→03Resolved [16:07:38] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196#12030197 (10aputhin) 05In progress→03Resolved [16:09:32] 10Toolforge, 06tools-platform-team: [image-config] deprecate and move all data to builds-api - https://phabricator.wikimedia.org/T409728#12030236 (10dcaro) [16:09:46] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#12030237 (10dcaro) [16:10:05] 06cloud-services-team, 10Data-Services, 10VPS-Projects, 13Patch-For-Review: Requesting access to NFS mount /public/dumps for language Cloud VPS project - https://phabricator.wikimedia.org/T429433#12030247 (10Volans) @santhosh as `mount_nfs: true` is set at project puppet level, the dumps NFS will be mounte... [16:12:43] 06tools-platform-team, 10Toolhub: Replace Bullseye base image with Trixie before August 2026 EOL deadline - https://phabricator.wikimedia.org/T425303#12030304 (10fnegri) a:03fnegri [16:16:00] 10wikitech.wikimedia.org: Requesting confirmed user access for User:Tejinderk.2004 - https://phabricator.wikimedia.org/T429511 (10TBurmeister) 03NEW [16:17:45] 06cloud-services-team, 10Toolforge, 07Documentation, 07good first task: Create a new doc about managing and sharing files in Toolforge - https://phabricator.wikimedia.org/T347753#12030378 (10TBurmeister) Hi! Sorry to hear you're encountering that AbuseFilter issue. I filed T429511 to request that your user... [16:19:31] 06cloud-services-team: cloudceph "HEALTH_WARN 17 OSD(s) experiencing slow operations in BlueStore" - https://phabricator.wikimedia.org/T429387#12030392 (10Andrew) Now running "ceph config set class:hdd bluestore_slow_ops_warn_threshold 10" [16:22:43] 06cloud-services-team: cloudceph "HEALTH_WARN 17 OSD(s) experiencing slow operations in BlueStore" - https://phabricator.wikimedia.org/T429387#12030413 (10Andrew) >>! In T429387#12028019, @fgiunchedi wrote: > Later in the thread there's a few suggested troubleshooting steps, how do those look? https://www.mail-a... [16:23:32] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [16:25:04] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: [toolforge-cli.build] Implement a --json flag to output machine-readable output - https://phabricator.wikimedia.org/T334589#12030428 (10dcaro) [16:25:12] 10Cloud Services Proposals, 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Push-to-Deploy), 06tools-platform-team, and 4 others: [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and ... - https://phabricator.wikimedia.org/T194332#12030429 [16:29:19] (03open) 10andrew: magnum: add cluster-api-based coe cluster. [repos/cloud/cloud-vps/tf-infra-test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tf-infra-test/-/merge_requests/8 (https://phabricator.wikimedia.org/T393782) [16:29:36] (03merge) 10andrew: magnum: add cluster-api-based coe cluster. [repos/cloud/cloud-vps/tf-infra-test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tf-infra-test/-/merge_requests/8 (https://phabricator.wikimedia.org/T393782) [16:35:13] 10wikitech.wikimedia.org: Requesting confirmed user access for User:Tejinderk.2004 - https://phabricator.wikimedia.org/T429511#12030480 (10JJMC89) Granting confirmed wouldn't have worked. I've updated the filter so that it will now. A Wikitech bureaucrat is needed to action this request. [16:38:32] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [16:43:40] 10wikitech.wikimedia.org: Requesting confirmed user access for User:Tejinderk.2004 - https://phabricator.wikimedia.org/T429511#12030504 (10aaron) 05Open→03Resolved a:03aaron [17:09:02] 10Tool-wikimedia-attribution, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Check Editor Counts - https://phabricator.wikimedia.org/T427548#12030584 (10AKhatun_WMF) The staging table(s) now have all revisions from `simplewiki` as of `2026-06-17 15:41:50... [17:15:34] 06cloud-services-team, 10Toolforge, 07Documentation, 07good first task: Create a new doc about managing and sharing files in Toolforge - https://phabricator.wikimedia.org/T347753#12030659 (10TBurmeister) @Tejinderk.2004 - your user rights have been adjusted. Please log out of Wikitech and log back in, then... [17:28:46] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Establish a blackbox network probe vantage point into cloud realm - https://phabricator.wikimedia.org/T429451#12030754 (10cmooney) @fgiunchedi it seems like a reasonable idea yeah.... [17:35:56] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering, 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12030801 (10Ottomata) [17:37:27] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering, 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12030805 (10Ahoelzl) p:05Triage→03High [17:41:29] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering, 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12030808 (10Ottomata) a:03Ottomata Assigning to myself for coordination. [17:44:23] 06cloud-services-team, 10Data-Services, 10VPS-Projects: Requesting access to NFS mount /public/dumps for language Cloud VPS project - https://phabricator.wikimedia.org/T429433#12030818 (10Volans) 05Open→03Resolved a:03Volans The dumps seems to be mounted and accessible, I'm tentatively resolving th... [18:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:43:02] (03update) 10fnegri: _update_storage_job_status_from_runtime: exclude image.exists/status [repos/cloud/toolforge/jobs-api] (fix_aliases) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/312 (owner: 10dcaro) [18:46:46] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12031139 (10fnegri) Unfortunately it has not stabilized and it's growing again: {F89161861} Current long transactions: `lang=mysql MariaDB [(none)]>... [18:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:29:19] 10Tool-wikimedia-attribution, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Check Editor Counts - https://phabricator.wikimedia.org/T427548#12031270 (10AKhatun_WMF) 1/ So I was looking at some pages, and wanted to note that we count "Editors", so revisi... [19:35:21] 06cloud-services-team, 10Toolforge, 07Documentation, 07good first task: Create a new doc about managing and sharing files in Toolforge - https://phabricator.wikimedia.org/T347753#12031320 (10Tejinderk.2004) Hi @apaskulin , @TBurmeister , and BLiviero-WMF, I've completed the task! This was my first Phabric... [20:15:32] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 85.4%) at host paws-nfs-1 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [20:29:29] 10Cloud-Services: debian-13.0-trixie image still references mirrors.wikimedia.org - https://phabricator.wikimedia.org/T429542 (10dancy) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a... [20:29:31] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 07Epic: [KR] WE6.3 Introduce a sustainability scoring system for the Toolforge platform - https://phabricator.wikimedia.org/T368600#12031508 (10komla) As shared, this will be revisited, but initial assessment over the three main categories of Obser... [20:29:48] 10Cloud-Services: debian-13.0-trixie image still references mirrors.wikimedia.org - https://phabricator.wikimedia.org/T429542#12031520 (10dancy) [20:48:00] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: move math-nfs-1 to new instance - https://phabricator.wikimedia.org/T429544 (10Physikerwelt) 03NEW [20:56:23] 10Cloud-Services: debian-12.0-bookworm and debian-13.0-trixie image still reference mirrors.wikimedia.org - https://phabricator.wikimedia.org/T429542#12031620 (10dancy) [21:00:25] 06cloud-services-team, 10Cloud-VPS: debian-12.0-bookworm and debian-13.0-trixie image still reference mirrors.wikimedia.org - https://phabricator.wikimedia.org/T429542#12031622 (10JJMC89) [21:29:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [21:34:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [22:55:33] 06cloud-services-team, 10Toolforge, 07Documentation, 07good first task: Create a new doc about managing and sharing files in Toolforge - https://phabricator.wikimedia.org/T347753#12031972 (10apaskulin) 05Open→03Resolved I reviewed the page, and it looks excellent. Great work! I've gone ahead and ad... [23:19:03] 10Cloud-Services, 06tools-infrastructure-team: Implement magnum monitoring - https://phabricator.wikimedia.org/T429557 (10BLiviero-WMF) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with... [23:19:56] FIRING: SystemdUnitDown: The service unit security_group_ssh-from-restricted-bastion_to_project_zuul.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [23:34:56] RESOLVED: SystemdUnitDown: The service unit security_group_ssh-from-restricted-bastion_to_project_zuul.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [23:49:56] FIRING: SystemdUnitDown: The service unit security_group_ssh-from-restricted-bastion_to_project_zuul.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown