[11:21:34] FIRING: DiskSpace: Disk space puppetboard1003:9100:/ 3.578% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=puppetboard1003 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:22:37] ^ that's caused by the new kernel image rolled out, I'll free some disk space [11:26:34] RESOLVED: DiskSpace: Disk space puppetboard1003:9100:/ 3.489% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=puppetboard1003 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:00:12] 10netops, 10Cloud-VPS, 10Data-Services, 06Infrastructure-Foundations, and 4 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12311975 (10Gehel) [15:23:18] 10CFSSL-PKI, 06Infrastructure-Foundations, 13Patch-For-Review: Add a load balanced service in front of PKI - https://phabricator.wikimedia.org/T436809#12312519 (10elukey) Reporting what has been discussed on IRC and Gerrit so far: * We already have an LVS config for pki.discovery.wmnet, plus a gdns configur... [15:26:15] 10CFSSL-PKI, 06Infrastructure-Foundations, 13Patch-For-Review: Add a load balanced service in front of PKI - https://phabricator.wikimedia.org/T436809#12312536 (10jhathaway) I agree with your suggestion to move signing to pki2002.codfw.wmnet. That alleviates the time pressure and noise during the operation.