[00:03:55] (MaxConntrack) firing: Max conntrack at 81.93% on cloudvirt1042:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:09:09] 10Toolforge: Buildservice "network is unreachable" error - https://phabricator.wikimedia.org/T362958#9732485 (10bd808) [00:15:44] 10Toolforge: Buildservice "network is unreachable" error - https://phabricator.wikimedia.org/T362958#9732498 (10bd808) The root problem here seems to be: `COUNTEREXAMPLE [step-build] 2024-04-19T08:31:50.877313171Z Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (1... [00:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:47:41] (CloudVPSDesignateLeaks) firing: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:48:55] (MaxConntrack) resolved: Max conntrack at 81.36% on cloudvirt1042:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:52:41] (CloudVPSDesignateLeaks) firing: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:57:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:59:06] 10Data-Services, 06DBA: High replication lag for enwiki (db1154 s1 replication crashed) - https://phabricator.wikimedia.org/T363077#9732505 (10Liz) Now a 7 hour system lag. [01:14:28] 10Data-Services, 06DBA: High replication lag for enwiki (db1154 s1 replication crashed) - https://phabricator.wikimedia.org/T363077#9732514 (10Pppery) FYI comments complaining about how the lag is getting longer etc. aren't helpful. Everyone knows it is getting worse, and it will continue to get worse until it... [01:41:01] 10Data-Services: Some PetScan queries do not return any results anymore for some days now - https://phabricator.wikimedia.org/T363073#9732516 (10Bugreporter) [01:44:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:54:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:40:54] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [03:49:23] 10Data-Services, 06DBA: High replication lag for enwiki (db1154 s1 replication crashed) - https://phabricator.wikimedia.org/T363077#9732548 (10Liz) Well, my "complaints" are actually a request for information on when this problem will be fixed. Where else can I ask? I was directed to come here. [07:14:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:24:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:45:54] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [07:57:22] 10Toolforge (Software install/update): Cant create virtualenv on Toolforge - https://phabricator.wikimedia.org/T363071#9732661 (10JJMC89) 05Open→03Invalid Documentation covering Toolforge job and webservice venvs: - [[ https://wikitech.wikimedia.org/wiki/Help:Toolforge/Python | Help:Toolforge/Python ]] -... [08:20:00] 10Data-Services, 06DBA: High replication lag for enwiki (db1154 s1 replication crashed) - https://phabricator.wikimedia.org/T363077#9732692 (10Aklapper) The comment included no question and no request. The problem will be fixed when this task status is set to "resolved" as regularly posting "someone is working... [09:59:56] 10Data-Services, 06DBA: High replication lag for enwiki (db1154 s1 replication crashed) - https://phabricator.wikimedia.org/T363077#9732808 (10Ladsgroup) FWIW, this is a data corruption issue. Last time it happened on sanitarium hosts the whole system went down for a week. I'm trying to figure out if I can avo... [10:39:51] 10Toolforge: /mnt/nfs/labstore-secondary-tools-project no longer seems to be mounted in the new container on Toolforge - https://phabricator.wikimedia.org/T363087 (10SunAfterRain) 03NEW [10:44:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:54:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:06:19] 10Data-Services, 06DBA: High replication lag for enwiki (db1154 s1 replication crashed) - https://phabricator.wikimedia.org/T363077#9732870 (10Wbm1058) Sorry to butt in, but, my understanding from the above is that the corrupted table coincidentally happens to be the same table which was just recently "normali... [11:15:24] 10Data-Services, 06DBA: High replication lag for enwiki (db1154 s1 replication crashed) - https://phabricator.wikimedia.org/T363077#9732873 (10Marostegui) >>! In T363077#9732808, @Ladsgroup wrote: > FWIW, this is a data corruption issue. Last time it happened on sanitarium hosts the whole system went down for... [11:17:42] 10Data-Services, 06DBA: High replication lag for enwiki (db1154 s1 replication crashed) - https://phabricator.wikimedia.org/T363077#9732875 (10Marostegui) Another option is to force a table rebuilt entirely if the above doesn't work: `alter table pagelinks engine=InnoDB,force;` [11:50:54] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [11:55:34] 06cloud-services-team, 10wikitech.wikimedia.org, 06Trust-and-Safety: Account recovery help needed for Developer account RLuts - https://phabricator.wikimedia.org/T363092 (10RLuts) 03NEW [12:00:00] 06cloud-services-team, 10wikitech.wikimedia.org, 06Trust-and-Safety: Account recovery help needed for Developer account RLuts - https://phabricator.wikimedia.org/T363092#9732970 (10RLuts) tools-bastion-13.tools.eqiad1.wikimedia.cloud:/home/rluts/2fa-reset-request.txt [12:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:22:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:24:26] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1023080 (owner: 10L10n-bot) [12:48:57] 10Data-Services, 06DBA: High replication lag for enwiki (db1154 s1 replication crashed) - https://phabricator.wikimedia.org/T363077#9733056 (10Ladsgroup) Thanks I will try that! [13:01:09] 10Data-Services, 06DBA: High replication lag for enwiki (db1154 s1 replication crashed) - https://phabricator.wikimedia.org/T363077#9733070 (10Marostegui) Worst case scenario, we can try to add a replication filter to skip that table, let replication catch up, and then reimport only that table from the master.... [13:02:10] 10Data-Services, 06DBA: High replication lag for enwiki (db1154 s1 replication crashed) - https://phabricator.wikimedia.org/T363077#9733071 (10Marostegui) With those two things I meant the: alter table to rebuild the table and the index drop+creation. The replication filter thing we can evaluate once the abov... [13:29:35] (DiskSpace) firing: Disk space cloudbackup1004:9100:/srv 5.961% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:51:30] 10Data-Services, 06DBA: High replication lag for enwiki (db1154 s1 replication crashed) - https://phabricator.wikimedia.org/T363077#9733124 (10Ladsgroup) >>! In T363077#9732470, @Ladsgroup wrote: > I started an optimize table on pagelinks, It's unlikely it would fix the issue but worth a try. This actually se... [13:56:17] 10Data-Services, 06DBA: High replication lag for enwiki (db1154 s1 replication crashed) - https://phabricator.wikimedia.org/T363077#9733125 (10Ladsgroup) >>! In T363077#9732870, @Wbm1058 wrote: > Sorry to butt in, but, my understanding from the above is that the corrupted table coincidentally happens to be the... [13:58:51] 10Data-Services, 06DBA: High replication lag for enwiki (db1154 s1 replication crashed) - https://phabricator.wikimedia.org/T363077#9733128 (10Ladsgroup) Anyway, the lag is decreasing, It'll take two hours or so to fully recover (unless there is another replication crash). I will close this once the lag goes b... [14:14:35] (DiskSpace) resolved: Disk space cloudbackup1004:9100:/srv 5.939% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:16:30] (03PS1) 10Majavah: releases: Bump jsdoc-wmf-theme [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1023097 (https://phabricator.wikimedia.org/T187672) [15:17:01] (03CR) 10Majavah: "Please include bumping the config here in whatever release process the package has." [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1023097 (https://phabricator.wikimedia.org/T187672) (owner: 10Majavah) [15:40:51] 10Tools, 10Maps (Maps-data): Set up a Nominatim instance to avoid geocode lookups with Google and/or OSM from tools - https://phabricator.wikimedia.org/T189879#9733321 (10DB111) 05Resolved→03Open Did Nominatim loose it's CORS header? Can no longer request from other toolforge tool. [15:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:47:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:52:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:55:54] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [15:57:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:19:57] 10Tools, 10Maps (Maps-data): Set up a Nominatim instance to avoid geocode lookups with Google and/or OSM from tools - https://phabricator.wikimedia.org/T189879#9733393 (10TheDJ) I did update it last week, ill take a look later tonight. Which toolforge tool btw ? [16:24:30] 10Tools, 10Maps (Maps-data): Set up a Nominatim instance to avoid geocode lookups with Google and/or OSM from tools - https://phabricator.wikimedia.org/T189879#9733398 (10DB111) https://wikimap.toolforge.org, thank you! [16:29:19] 06cloud-services-team, 10wikitech.wikimedia.org, 06Trust-and-Safety: Account recovery help needed for Developer account RLuts - https://phabricator.wikimedia.org/T363092#9733409 (10bd808) 05Open→03Resolved a:03bd808 `lang=shell-session $ sudo ls -lh /home/rluts/2fa-reset-request.txt -rw------- 1 rl... [17:28:36] (03CR) 10VolkerE: [C:03+2] releases: Bump jsdoc-wmf-theme [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1023097 (https://phabricator.wikimedia.org/T187672) (owner: 10Majavah) [17:29:11] (03Merged) 10jenkins-bot: releases: Bump jsdoc-wmf-theme [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1023097 (https://phabricator.wikimedia.org/T187672) (owner: 10Majavah) [17:36:42] 10Data-Services, 06DBA: High replication lag for enwiki (db1154 s1 replication crashed) - https://phabricator.wikimedia.org/T363077#9733523 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup The rebuild fixed the issue. The replag is zero. [18:05:57] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1023080 (owner: 10L10n-bot) [18:43:23] 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10Temporary accounts, 10XTools, and 2 others: [Design] Update wireframes with user testing learnings - https://phabricator.wikimedia.org/T359827#9733606 (10KColeman-WMF) 05Open→03Resolved [18:44:52] 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10XTools, 07Design, and 2 others: [Design] Communicate to users why there are gaps in IP data on Special:Contributions - https://phabricator.wikimedia.org/T360928#9733605 (10KColeman-WMF) [18:55:19] 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10Temporary accounts, 10XTools, and 2 others: [Design] UI design for GUC - https://phabricator.wikimedia.org/T363115 (10KColeman-WMF) 03NEW [18:58:02] 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10Temporary accounts, 10XTools, 07Design: [Design EPIC] Global User Contributions - https://phabricator.wikimedia.org/T349901#9733624 (10KColeman-WMF) [19:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:17:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:22:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:27:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:31:38] 10Tools, 10Maps (Maps-data): Set up a Nominatim instance to avoid geocode lookups with Google and/or OSM from tools - https://phabricator.wikimedia.org/T189879#9733646 (10TheDJ) solved by restarting the server. Next time, please open a NEW ticket for NEW problems. you can always link back to a ticket like this... [19:34:29] 10Tools, 10Maps (Maps-data): Set up a Nominatim instance to avoid geocode lookups with Google and/or OSM from tools - https://phabricator.wikimedia.org/T189879#9733647 (10TheDJ) 05Open→03Resolved [20:00:54] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [20:12:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:16:56] 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10Temporary accounts, 07Design: [Design EPIC] Global User Contributions - https://phabricator.wikimedia.org/T349901#9733730 (10MusikAnimal) [20:16:59] 10Tool-Global-user-contributions: Add Flow contributions to GUC - https://phabricator.wikimedia.org/T114777#9733731 (10Krinkle) [20:17:20] 10Tool-Global-user-contributions: Edits in the Topic: namespace don't show in GUC - https://phabricator.wikimedia.org/T343790#9733733 (10Krinkle) →14Duplicate dup:03T114777 [20:18:26] 10Tool-Global-user-contributions: Add Flow contributions to GUC - https://phabricator.wikimedia.org/T114777#9733735 (10Krinkle) [20:19:36] 10Tool-Global-user-contributions: GUC should make it more intuitive that entries are limited by wiki - https://phabricator.wikimedia.org/T259454#9733749 (10Krinkle) [20:19:51] 10Tool-Global-user-contributions: GUC should make it more intuitive that entries are limited by wiki - https://phabricator.wikimedia.org/T259454#9733750 (10Krinkle) p:05Triage→03Medium [20:20:09] 10Tool-Global-user-contributions: GUC unreachable - https://phabricator.wikimedia.org/T351194#9733745 (10Krinkle) 05Open→03Resolved a:03Krinkle This was due to general maintenance with Toolforge, not specific to GUC. [20:22:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:17:41] (CloudVPSDesignateLeaks) firing: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:22:41] (CloudVPSDesignateLeaks) firing: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:27:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:17:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks