[00:33:51] 10VPS-project-Codesearch, 06Data-Platform, 10Dumps-Generation: Index operations/dumps - https://phabricator.wikimedia.org/T378863 (10Milimetric) 03NEW [01:22:36] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:22:36] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:22:36] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:38:29] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-idp-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:22:36] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:52:39] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate deployment-poolcounter06.deployment-prep.eqiad.wmflabs is about to expire in 25d 23h 58m 30s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [14:03:12] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-test-k8s-haproxy-6 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [17:22:36] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:07:25] 06cloud-services-team, 10Toolforge, 07Wikimedia-production-error: mysqldump Command Not Found on tools Bastion - https://phabricator.wikimedia.org/T378882 (10Gopavasanth) 03NEW p:05Triage→03Medium [21:08:28] 06cloud-services-team, 10Toolforge, 07Wikimedia-production-error: mysqldump Command Not Found on tools Bastion - https://phabricator.wikimedia.org/T378882#10286481 (10Gopavasanth) [21:18:12] FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-harbor-1 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:18:32] 06cloud-services-team, 10Toolforge: mysqldump Command Not Found on tools Bastion - https://phabricator.wikimedia.org/T378882#10286482 (10Pppery) [21:22:36] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:30:16] 06cloud-services-team, 10Toolforge: mysqldump Command Not Found on tools Bastion - https://phabricator.wikimedia.org/T378882#10286494 (10JJMC89) p:05Medium→03Triage [22:35:28] 10Tool-toolwatch: Toolwatch incorrectly reports non-web tools as unavailable - https://phabricator.wikimedia.org/T376451#10286502 (10Gopavasanth) 05Open→03Resolved Changes have been successfully deployed! @JJMC89, please let us know if the behavior has improved or remains the same in the future. You can... [22:41:23] 06cloud-services-team, 10Toolforge: mysqldump Command Not Found on tools Bastion - https://phabricator.wikimedia.org/T378882#10286506 (10Soda) If not `mysqldump`, at least `mariadb-dump` should be included. [22:43:59] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10286507 (10Gopavasanth) @Soylacarli, @Fnielsen, @Nux, thank you for highlighting the bugs and gaps in the tool! The issues have been identified, fixed, and deployed. Please let us know if th... [22:57:14] 10Tool-toolwatch: Toolwatch incorrectly reports non-web tools as unavailable - https://phabricator.wikimedia.org/T376451#10286514 (10JJMC89) 05Resolved→03Open The tool linked in the description is still incorrectly reported as unavailable. [23:16:20] 10Tool-toolwatch: Toolwatch incorrectly reports non-web tools as unavailable - https://phabricator.wikimedia.org/T376451#10286518 (10Gopavasanth) Aah @JJMC89, this tool is specifically designed to monitor the status of web-based tools; it currently does not track non-web-based tools. So if you look at the stats... [23:45:00] 10Tool-toolwatch: Toolwatch incorrectly reports non-web tools as unavailable - https://phabricator.wikimedia.org/T376451#10286521 (10JJMC89) The non-web tools are still listed as unavailable though. They should be removed. {F57672130}