[00:05:54] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [00:54:55] (MaxConntrack) firing: Max conntrack at 80.52% on cloudvirt1042:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:59:55] (MaxConntrack) resolved: Max conntrack at 80.29% on cloudvirt1042:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [01:14:56] 10Tool-bridgebot: Replace custom deployment with build service and job service - https://phabricator.wikimedia.org/T363028#9733917 (10bd808) I recently learned that matterbridge uses the https://github.com/spf13/viper library and its `AutomaticEnv` feature when processing the config file. This allows envvars lik... [03:17:41] (CloudVPSDesignateLeaks) firing: (3) Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:28:31] 10Toolforge: [builds-builder] Support using custom buildpacks - https://phabricator.wikimedia.org/T363033#9734020 (10bd808) Adding a buildpack like https://github.com/kr/heroku-buildpack-inline to our stack might be an interesting way to implement this feature request. That solution would not readily support the... [03:54:59] 06cloud-services-team, 10WMF-General-or-Unknown, 07Epic, 07Security: sustainability of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T363125 (10Andrew) 03NEW [03:56:36] 06cloud-services-team, 10wikitech.wikimedia.org, 07LDAP: Replace wikitech as source of two-factor auth protection for developer accounts - https://phabricator.wikimedia.org/T359551#9734063 (10Andrew) [03:56:37] 06cloud-services-team, 10WMF-General-or-Unknown, 07Epic, 07Security: sustainability of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T363125#9734062 (10Andrew) [03:56:39] 06cloud-services-team, 10wikitech.wikimedia.org: Disable SSH key management on Wikitech - https://phabricator.wikimedia.org/T359544#9734064 (10Andrew) [03:56:39] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#9734065 (10Andrew) [03:56:41] 06cloud-services-team, 10wikitech.wikimedia.org: Move Wikitech onto the production MW cluster - https://phabricator.wikimedia.org/T237773#9734066 (10Andrew) [03:56:45] 06cloud-services-team, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#9734068 (10Andrew) [03:56:47] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#9734067 (10Andrew) [04:10:54] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [04:22:37] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic, 07Security: sustainability of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T363125#9734073 (10Pppery) [05:26:46] 14Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9734075 (10MBH) @dcaro A Visual Studio currently provides two types of console C# apps: old way, working on Windows only (or under mono), and a new "dotnet" way, claimed t... [06:50:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance cloudinfra-acme-chief-02 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [06:53:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance runner-1029 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [06:54:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance proxy-04 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [06:55:28] (PuppetAgentNoResources) firing: (2) No Puppet resources found on instance cloudinfra-acme-chief-02 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [06:59:28] (PuppetAgentNoResources) firing: (2) No Puppet resources found on instance maps-proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:03:28] (PuppetAgentNoResources) firing: (4) No Puppet resources found on instance runner-1025 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:04:28] (PuppetAgentNoResources) firing: (3) No Puppet resources found on instance maps-proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:05:28] (PuppetAgentNoResources) firing: (3) No Puppet resources found on instance cloudinfra-acme-chief-02 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:06:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance cvn-app10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:06:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance extdist-06 on project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:07:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance tf-bastion on project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:08:28] (PuppetAgentNoResources) firing: (6) No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:09:28] (PuppetAgentNoResources) firing: (2) No Puppet resources found on instance toolsbeta-test-k8s-control-7 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:09:28] (PuppetAgentNoResources) firing: (4) No Puppet resources found on instance tools-db-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:09:28] (PuppetAgentNoResources) firing: (5) No Puppet resources found on instance maps-proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:09:32] (PuppetAgentNoResources) firing: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:10:28] (PuppetAgentNoResources) firing: (7) No Puppet resources found on instance cloudinfra-acme-chief-02 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:10:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance metricsinfra-puppetserver-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:11:28] (PuppetAgentNoResources) firing: (3) No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:13:28] (PuppetAgentNoResources) firing: (7) No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:14:28] (PuppetAgentNoResources) firing: (8) No Puppet resources found on instance toolsbeta-harbor-1 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:14:28] (PuppetAgentNoResources) firing: (25) No Puppet resources found on instance tools-cumin-1 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:15:28] (PuppetAgentNoResources) firing: (8) No Puppet resources found on instance cloud-cumin-04 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:17:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance clouddb-services-puppetserver-1 on project clouddb-services - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:17:40] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T363131#9734172 (10LibUp-bot) [07:17:40] 10Toolforge: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T363132#9734174 (10LibUp-bot) [07:17:41] (CloudVPSDesignateLeaks) firing: (3) Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:18:28] (PuppetAgentNoResources) firing: (11) No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:19:28] (PuppetAgentNoResources) firing: (6) No Puppet resources found on instance maps-proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:19:28] (PuppetAgentNoResources) firing: (40) No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:19:28] (PuppetAgentNoResources) firing: (13) No Puppet resources found on instance toolsbeta-docker-imagebuilder-2 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:19:32] (PuppetAgentNoResources) firing: (2) No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:20:28] (PuppetAgentNoResources) firing: (15) No Puppet resources found on instance cloud-cumin-03 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:21:28] (PuppetAgentNoResources) firing: (3) No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:21:28] (PuppetAgentNoResources) resolved: No Puppet resources found on instance extdist-06 on project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:21:56] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic, 07Security: sustainability of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T363125#9734178 (10taavi) > B: Fishbowl wiki hosted on wikikube, accounts in ldap. This option could be a final state OR a temporary state on the way to the SUL... [07:22:28] (PuppetAgentNoResources) resolved: No Puppet resources found on instance tf-bastion on project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:23:28] (PuppetAgentNoResources) firing: (11) No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:24:28] (PuppetAgentNoResources) firing: (18) No Puppet resources found on instance toolsbeta-cumin-1 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:24:28] (PuppetAgentNoResources) firing: (51) No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:25:28] (PuppetAgentNoResources) firing: (17) No Puppet resources found on instance cloud-cumin-03 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:25:28] (PuppetAgentNoResources) firing: (2) No Puppet resources found on instance metricsinfra-grafana-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:26:28] (PuppetAgentNoResources) firing: (4) No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:27:40] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic, 07Security: sustainability of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T363125#9734222 (10taavi) [07:27:42] 06cloud-services-team, 10Striker, 10Data-Persistence-Backup, 06DBA, 13Patch-For-Review: Create a database for Striker test instance - https://phabricator.wikimedia.org/T360149#9734212 (10ABran-WMF) 05Open→03Resolved I'm closing this @taavi, feel free to ping me if you run in any trouble! [07:29:28] (PuppetAgentNoResources) firing: (70) No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:29:28] (PuppetAgentNoResources) firing: (24) No Puppet resources found on instance toolsbeta-cumin-1 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:29:28] (PuppetAgentNoResources) firing: (2) No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:32:28] (PuppetAgentNoResources) resolved: No Puppet resources found on instance clouddb-services-puppetserver-1 on project clouddb-services - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:34:28] (PuppetAgentNoResources) firing: (88) No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:34:28] (PuppetAgentNoResources) firing: (33) No Puppet resources found on instance toolsbeta-acme-chief-2 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:34:28] (PuppetAgentNoResources) firing: (6) No Puppet resources found on instance maps-proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:34:32] (PuppetAgentNoResources) resolved: (2) No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:35:28] (PuppetAgentNoResources) firing: (17) No Puppet resources found on instance cloud-cumin-03 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:35:28] (PuppetAgentNoResources) firing: (4) No Puppet resources found on instance metricsinfra-grafana-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:39:28] (PuppetAgentNoResources) firing: (90) No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:39:28] (PuppetAgentNoResources) firing: (33) No Puppet resources found on instance toolsbeta-acme-chief-2 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:40:28] (PuppetAgentNoResources) firing: (17) No Puppet resources found on instance cloud-cumin-03 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:40:28] (PuppetAgentNoResources) firing: (8) No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:41:28] (PuppetAgentNoResources) resolved: (3) No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:44:28] (PuppetAgentNoResources) firing: (31) No Puppet resources found on instance toolsbeta-acme-chief-2 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:44:28] (PuppetAgentNoResources) firing: (86) No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:44:58] (PuppetAgentNoResources) firing: (2) No Puppet resources found on instance paws-nfs-1 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:45:28] (PuppetAgentNoResources) firing: (8) No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:49:28] (PuppetAgentNoResources) firing: (63) No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:49:28] (PuppetAgentNoResources) resolved: (25) No Puppet resources found on instance toolsbeta-acme-chief-2 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:50:28] (PuppetAgentNoResources) firing: (16) No Puppet resources found on instance cloud-cumin-03 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:50:28] (PuppetAgentNoResources) firing: (9) No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:54:28] (PuppetAgentNoResources) resolved: (50) No Puppet resources found on instance tools-acme-chief-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:54:28] (PuppetAgentNoResources) firing: (5) No Puppet resources found on instance maps-proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:55:28] (PuppetAgentNoResources) firing: (11) No Puppet resources found on instance cloud-cumin-03 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:55:28] (PuppetAgentNoResources) firing: (9) No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:55:58] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic, 07Security: sustainability of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T363125#9734253 (10MoritzMuehlenhoff) >>! In T363125#9734178, @taavi wrote: >> B: Fishbowl wiki hosted on wikikube, accounts in ldap. This option could be a fina... [07:59:58] (PuppetAgentNoResources) resolved: No Puppet resources found on instance paws-nfs-1 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:00:28] (PuppetAgentNoResources) firing: (9) No Puppet resources found on instance cloud-cumin-03 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:00:28] (PuppetAgentNoResources) firing: (9) No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:05:28] (PuppetAgentNoResources) resolved: (9) No Puppet resources found on instance cloud-cumin-03 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:05:28] (PuppetAgentNoResources) firing: (9) No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:09:28] (PuppetAgentNoResources) firing: (4) No Puppet resources found on instance maps-proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:10:28] (PuppetAgentNoResources) resolved: (6) No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:14:28] (PuppetAgentNoResources) resolved: (2) No Puppet resources found on instance maps-proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:15:54] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [08:26:26] 10Cloud Services Proposals: Decision request - Update python team best practices - https://phabricator.wikimedia.org/T361804#9734320 (10dcaro) >>! In T361804#9727897, @Andrew wrote: > I'm late to this, but I also agree with B5. Do we need another rule regarding what to do with existing code when applying new... [08:31:40] 10Cloud Services Proposals: Decision request - Update python team best practices - https://phabricator.wikimedia.org/T361804#9734346 (10dcaro) Created the page here https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Coding [08:32:55] 10Cloud Services Proposals: Decision request - Update python team best practices - https://phabricator.wikimedia.org/T361804#9734374 (10dcaro) >>! In T361804#9734320, @dcaro wrote: >>>! In T361804#9727897, @Andrew wrote: >> I'm late to this, but I also agree with B5. Do we need another rule regarding what to... [09:10:55] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2002-dev.codfw.wmnet' [09:15:50] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2002-dev.codfw.wmnet' [09:38:14] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9734666 (10aborrero) >>! In T362872#9726419, @fnegri wrote: > >> What do we do with the already defined Pod? > > This is something to verif... [09:40:22] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9734678 (10aborrero) >>! In T362872#9734666, @aborrero wrote: >>>! In T362872#9726419, @fnegri wrote: >> >>> What do we do with the already... [09:42:07] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9734680 (10taavi) >>! In T362872#9734678, @aborrero wrote: > In both cases, they don't mention anything about mutation. I assume this means t... [09:45:00] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9734693 (10aborrero) >>! In T362872#9734680, @taavi wrote: >>>! In T362872#9734678, @aborrero wrote: >> In both cases, they don't mention any... [09:47:24] 14Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9734708 (10dcaro) @MBH dotnet is enabled only through buildpacks, similar to what we did in https://github.com/Saisengen/wikibots. That means that the process is not:... [09:51:59] 14Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9734720 (10MBH) I have. But how to run such bots in buildpacks? Now I use `jobs.yaml` and specify path to exe file, how to do it with buildpacks? And why dotnet inaccessi... [10:14:17] 14Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9734870 (10dcaro) >>! In T319883#9734720, @MBH wrote: > I have. But how to run such bots in buildpacks? If you have a single binary from the source code, you can follow... [10:32:18] 06cloud-services-team, 10Toolforge: lima-kilo: replicate sssd setup from Toolforge - https://phabricator.wikimedia.org/T362966#9734931 (10aborrero) Patch https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/119 seems to work as expected. [11:13:20] 14Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9735085 (10MBH) Bot example: https://github.com/Saisengen/wikibots/blob/main/other-bots/checking-new-edits.cs It creates one exe file. [11:15:49] 14Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9735091 (10MBH) How to specify, should run be one-time, regular or bot should be online permanently? [11:17:41] (CloudVPSDesignateLeaks) firing: (3) Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:07:48] 14Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9735215 (10dcaro) >>! In T319883#9735085, @MBH wrote: > Bot example: https://github.com/Saisengen/wikibots/blob/main/other-bots/checking-new-edits.cs > It creates one exe... [12:11:07] 14Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9735225 (10dcaro) An example of entry in `jobs.yaml` for it: ` - name: checking-new-edits command: checking-new-edits image: tool-mbh/tool-mbh:latest mount: all #... [12:20:54] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [12:24:32] 10Cloud Services Proposals, 10cloud-services-team (FY2023/2024-Q3-Q4): Decision Request - Incident Response Process - https://phabricator.wikimedia.org/T348887#9735331 (10fnegri) The "dry run" is scheduled for April, 29th at 15:00 UTC. The plan is to simulate an incident by shutting down one or more codfw serv... [12:50:22] 10Cloud Services Proposals, 10cloud-services-team (FY2023/2024-Q3-Q4): Decision Request - Incident Response Process - https://phabricator.wikimedia.org/T348887#9735409 (10fnegri) 05In progress→03Stalled [13:21:38] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic, 07Security: sustainability of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T363125#9735541 (10Andrew) >>! In T363125#9734178, @taavi wrote: >> B: Fishbowl wiki hosted on wikikube, accounts in ldap. This option could be a final state OR... [13:32:44] (InterfaceSpeedError) firing: brq05a5494a-18 on cloudvirt2001-dev:9100 has the wrong speed: 1.25e+06. - https://wikitech.wikimedia.org/wiki/Monitoring/check_eth - https://grafana.wikimedia.org/d/000000562 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceSpeedError [13:32:54] 06cloud-services-team: InterfaceSpeedError brq05a5494a-18 on cloudvirt2001-dev:9100 has the wrong speed: 1.25e+06. - https://phabricator.wikimedia.org/T363164 (10phaultfinder) 03NEW [13:32:59] 10Cloud Services Proposals: Decision request - Update python team best practices - https://phabricator.wikimedia.org/T361804#9735595 (10Andrew) >>! In T361804#9734320, @dcaro wrote: >>>! In T361804#9727897, @Andrew wrote: >> I'm late to this, but I also agree with B5. Do we need another rule regarding what t... [13:38:32] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic, 07Security: sustainability of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T363125#9735615 (10MoritzMuehlenhoff) >>! In T363125#9735541, @Andrew wrote: > I started out thinking that but I don't think it's so easy to switch to RO ldap. T... [14:02:02] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic, 07Security: sustainability of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T363125#9735700 (10taavi) Password reset have been disabled for a few months now, email addresses since I deployed that patch last week, and I just sent https://... [14:19:13] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.25 - https://phabricator.wikimedia.org/T316107#9735804 (10dcaro) [14:19:27] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26 - https://phabricator.wikimedia.org/T327025#9735806 (10dcaro) [14:20:14] 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#9735808 (10dcaro) [14:22:41] (CloudVPSDesignateLeaks) firing: (3) Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:27:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:58:56] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 09): [prometheus] [grafana] set scrape interval in data source config - https://phabricator.wikimedia.org/T363176 (10fnegri) 03NEW [15:00:29] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 09): [prometheus] [grafana] set scrape interval in data source config - https://phabricator.wikimedia.org/T363176#9735949 (10fnegri) 05Open→03In progress [15:04:06] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Taavi knowledge transfer: Toolforge k8s upgrades - https://phabricator.wikimedia.org/T362445#9735966 (10dcaro) Notes: ` Apr 23, 2024 | Toolforge - Kubernetes Attendees: Andrew Bogott Arturo Borrero David Caro Raymond Olisaemeka Taavi Väänänen Slavina Stef... [15:14:09] 06cloud-services-team: InterfaceSpeedError brq05a5494a-18 on cloudvirt2001-dev:9100 has the wrong speed: 1.25e+06. - https://phabricator.wikimedia.org/T363164#9735996 (10Andrew) This is taavi messing with ovs [15:58:41] (PrometheusRestarted) firing: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [16:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:22:00] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [16:22:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:23:41] (PrometheusRestarted) resolved: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [16:34:35] RECOVERY - Disk space on cloudbackup1002-dev is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbackup1002-dev&var-datasource=eqiad+prometheus/ops [16:41:19] 10Tools, 06Tech-Docs-Team, 07Documentation, 03Wikimedia-Hackathon-2024: [Hackathon 2024] Improve technical documentation of tools - https://phabricator.wikimedia.org/T358040#9736536 (10TBurmeister) [16:42:37] 10Tools, 06Tech-Docs-Team, 07Documentation, 03Wikimedia-Hackathon-2024: [Hackathon 2024] Improve technical documentation of tools - https://phabricator.wikimedia.org/T358040#9736558 (10TBurmeister) I've created a set of instructions for how to use Toolhub to find and review tool documentation, and then use... [16:43:31] 06cloud-services-team, 13Patch-For-Review: Migrate cloudweb, cloudbackup, cloudmetrics physical servers off buster - https://phabricator.wikimedia.org/T332400#9736551 (10Andrew) [16:45:27] 06cloud-services-team, 13Patch-For-Review: Migrate cloudweb, cloudbackup, cloudmetrics physical servers off buster - https://phabricator.wikimedia.org/T332400#9736565 (10Andrew) It is safe to reimage cloudbackup1003 on April 30. [16:52:59] 10Tool-Global-user-contributions, 10mediawiki-core-hackathon-2024, 07patch-welcome: Edit summaries of wikibase edits are displayed incorrectly - https://phabricator.wikimedia.org/T293837#9736619 (10larissagaulia) [17:32:44] (InterfaceSpeedError) firing: brq05a5494a-18 on cloudvirt2001-dev:9100 has the wrong speed: 1.25e+06. - https://wikitech.wikimedia.org/wiki/Monitoring/check_eth - https://grafana.wikimedia.org/d/000000562 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceSpeedError [17:56:46] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services, 13Patch-For-Review: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964#9736944 (10Dzahn) >>! In T360964#9718714, @hashar wrote: > `gerrit-prod-1001.devtools.eqiad1.wikimedia.cloud` can be deleted: was shut... [18:42:09] 06cloud-services-team, 10VPS-Projects, 13Patch-For-Review, 10Puppet (Puppet 7.0): Migrate per-project Puppet servers to Puppet 7 - https://phabricator.wikimedia.org/T351452#9737121 (10Andrew) [19:10:03] 06cloud-services-team, 10VPS-Projects, 10fundraising-tech-ops, 10Puppet (Puppet 7.0): Update puppet civicrm-prototype puppetmaster - https://phabricator.wikimedia.org/T361595#9737209 (10Dwisehaupt) [19:26:16] 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): Update puppet wikidata-query puppetmaster - https://phabricator.wikimedia.org/T361596#9737327 (10Andrew) 05Open→03Resolved a:03Andrew I've built a new puppetserver in this project, wdqspuppetserver-1. Nothing was using the old one so pro... [19:27:50] 06cloud-services-team, 10VPS-Projects, 13Patch-For-Review, 10Puppet (Puppet 7.0): Migrate per-project Puppet servers to Puppet 7 - https://phabricator.wikimedia.org/T351452#9737346 (10Andrew) [19:28:22] 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): Update mailman project puppetmaster - https://phabricator.wikimedia.org/T361371#9737343 (10Andrew) 05Open→03Resolved a:03Andrew [19:28:34] 06cloud-services-team, 10VPS-Projects, 13Patch-For-Review, 10Puppet (Puppet 7.0): Migrate per-project Puppet servers to Puppet 7 - https://phabricator.wikimedia.org/T351452#9737351 (10Andrew) [19:28:47] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 05Goal, 13Patch-For-Review, 10Puppet (Puppet 7.0): Migrate Cloud VPS puppet infrastructure to Puppet 7 - https://phabricator.wikimedia.org/T351450#9737352 (10Andrew) 05Open→03Resolved [19:29:41] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643#9737354 (10Andrew) a:05Andrew→03dcaro [19:43:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:53:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:03:05] 10Tool-logo-test: logo-test is down - https://phabricator.wikimedia.org/T363216 (10taavi) 03NEW [20:15:37] 10Tool-logo-test, 13Patch-For-Review: logo-test is down - https://phabricator.wikimedia.org/T363216#9737542 (10CodeReviewBot) taavi opened https://gitlab.wikimedia.org/toolforge-repos/logo-test/-/merge_requests/2 Remove duplicate healthz route [20:22:00] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:12:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:22:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:06:47] 10Data-Services, 06DBA: Prepare and check storage layer for bewwiki - https://phabricator.wikimedia.org/T360309#9738439 (10Zabe) wiki has been created [22:31:33] 10Data-Services, 06DBA: Prepare and check storage layer for kuswiki - https://phabricator.wikimedia.org/T360302#9738544 (10Zabe) wiki has been created [22:41:29] 10Data-Services, 06DBA: Prepare and check storage layer for mywikisource - https://phabricator.wikimedia.org/T363269#9738578 (10Zabe) wiki has been created [22:44:55] 10Toolforge, 10Tools, 06Data-Engineering, 10EventStreams, and 2 others: Frequent `429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange` errors in SULWatcher - https://phabricator.wikimedia.org/T329327#9738589 (10bd808) [22:52:42] 10Data-Services, 06DBA: Prepare and check storage layer for iglwiki - https://phabricator.wikimedia.org/T363262#9738608 (10Zabe) wiki has been created [23:12:49] 10VPS-project-Wikistats: Add iglwiki to wikistats - https://phabricator.wikimedia.org/T363267#9738681 (10Dzahn) 05Open→03Resolved a:03Dzahn ` MariaDB [wikistats]> insert into wikipedias (prefix, lang, loclang, method) values ("igl", "Igala", "Igala", 8); Query OK, 1 row affected (0.008 sec) dzahn@wiki... [23:14:47] 10Data-Services, 06DBA: Prepare and check storage layer for kaawiktionary - https://phabricator.wikimedia.org/T363255#9738686 (10Zabe) wiki has been created [23:16:51] 10VPS-project-Wikistats: Add kaawiktionary to wikistats - https://phabricator.wikimedia.org/T363260#9738692 (10Dzahn) 05Open→03Resolved a:03Dzahn ` MariaDB [wikistats]> insert into wiktionaries (prefix,lang,loclang,loclanglink,method) values ("kaa","Karakalpak","Qaraqalpaqsha","Qaraqalpaq tili",8); ..... [23:27:24] 10VPS-project-Wikistats: Add kuswiki to wikistats - https://phabricator.wikimedia.org/T360307#9738727 (10Dzahn) 05Stalled→03Resolved ` MariaDB [wikistats]> insert into wikipedias (prefix, lang, loclang, method) values ("kus", "Kusaal", "Kusaal", 8); ... dzahn@wikistats-bookworm:~$ /usr/lib/wikistats/up... [23:32:23] 10VPS-project-Wikistats: Add bewwiki to wikistats - https://phabricator.wikimedia.org/T360314#9738738 (10Dzahn) 05Stalled→03Resolved ` MariaDB [wikistats]> insert into wikipedias (prefix, lang, loclang, method) values ("bew", "Betawi", "Betawi", 8); ... dzahn@wikistats-bookworm:~$ /usr/lib/wikistats/upd... [23:35:45] (03PS1) 10AntiCompositeNumber: SULWatcher: add PingServer mixin to handle ping timeouts [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1023548 [23:41:05] 10Data-Services, 06DBA: Prepare and check storage layer for mswikisource - https://phabricator.wikimedia.org/T363249#9738763 (10Zabe) wiki has been created [23:41:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:43:25] 10Data-Services, 06DBA: Prepare and check storage layer for kawikisource - https://phabricator.wikimedia.org/T363242#9738767 (10Zabe) wiki has been created [23:46:22] 10VPS-project-Wikistats: Add kawikisource to wikistats - https://phabricator.wikimedia.org/T363247#9738769 (10Dzahn) a:03Dzahn [23:46:52] 10VPS-project-Wikistats: Add mswikisource to wikistats - https://phabricator.wikimedia.org/T363253#9738770 (10Dzahn) a:03Dzahn [23:47:29] 06cloud-services-team, 10VPS-Projects, 10fundraising-tech-ops, 10Puppet (Puppet 7.0): Update puppet civicrm-prototype puppetmaster - https://phabricator.wikimedia.org/T361595#9738771 (10Dwisehaupt) Thanks for the followup in IRC. We are migrating the service to the Prod VPS in the next week. After that, I'... [23:56:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks