[03:19:58] FIRING: [2x] SystemdUnitFailed: xfs_scrub_all.service on ms-be1090:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:19:58] FIRING: [2x] SystemdUnitFailed: xfs_scrub_all.service on ms-be1090:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:40:38] jhathaway: sorry, I was OOO, I will take a look [08:53:55] FIRING: [3x] SystemdUnitFailed: prometheus-mysqld-exporter.service on db1269:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:54:24] fixed ^ [08:58:55] FIRING: [3x] SystemdUnitFailed: prometheus-mysqld-exporter.service on db1269:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:09:49] elukey: I dunno if you went ahead while I was OoO, but no worries from here. You're on about 1.4T currently. [09:09:59] Emperor: there is a question in #mediawiki about an SVG on https://en.wikipedia.org/wiki/Soviet_empire not working, I can reproduce for https://en.wikipedia.org/wiki/File:Warsaw_Pact_in_1990_(orthographic_projection).svg. I purged the page but it doesn't seem to have fully worked. Is that a swift or a mediawiki error? It's just giving 400 Bad Request [09:10:01] Request served via cp3081 cp3081, Varnish XID 129534545 [09:10:01] 10:05:37 [09:10:01] Upstream caches: cp3081 int [09:10:01] 10:05:37 [09:10:01] Error: 400, Bad Request at Mon, 31 Aug 2026 09:04:51 GMT [09:11:20] Emperor: I did it in batches, added ~300GB, so I proceeded since it seemed harmless! :) [09:11:33] https://upload.wikimedia.org/wikipedia/commons/a/a8/Warsaw_Pact_in_1990_%28orthographic_projection%29.svg?utm_source=en.wikipedia.org&utm_campaign=index&utm_content=original is working for me right now. If there's an issue with it please open a ticket, I have a pile of things to get back to after 3 days off, and some of them are higher priority. [09:17:06] Emperor: https://phabricator.wikimedia.org/T436505 [09:18:24] ta [12:58:55] FIRING: [2x] SystemdUnitFailed: xfs_scrub_all.service on ms-be1090:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:18:55] FIRING: [3x] SystemdUnitFailed: swift-object.service on ms-be1090:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:22:52] ^-- currently attempting xfs_repair on the sad partition (if/when that fails I'll give up on the disk) [14:44:49] FIRING: PuppetFailure: Puppet has failed on ms-be1090:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:58:55] FIRING: [4x] SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:36:11] I'm beginning to think rclone has a bug handling paths with ‛ in [15:38:07] Emperor low priority, but I was looking at https://phabricator.wikimedia.org/T432944 (proposing to host static files on thanos-swift or apus) which was declined. I'm following up with DPE but just wondering if you had any additional context on why thanos-swift/apus was ruled out [15:39:30] FIRING: [4x] SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:52:06] inflatador: there's meeting minutes and everything; I'm afraid I'm busy for the next hour and a bit and then it's the end of my working day, but I can try and fish the relevant docs out for you tomorrow [15:52:44] Emperor sure, no hurry whatsoever [15:53:18] inflatador: found the handy summary on Slack, forwarded it on to you there. I hope that's OK [15:56:18] Emperor yup, this is more than enough context, thanks again [15:57:58] cool [16:04:49] RESOLVED: PuppetFailure: Puppet has failed on ms-be1090:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [19:39:58] FIRING: [3x] SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed