[07:23:57] <jinxer-wm>	 FIRING: SystemdUnitFailed: pt-heartbeat-wikimedia.service on db1268:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:28:50] <marostegui>	 ^expected
[08:18:57] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: pt-heartbeat-wikimedia.service on db1268:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:20:12] <jynus>	 being setup?
[08:20:58] <marostegui>	 yes
[08:21:32] <jynus>	 Let me know if you need help, e.g. provisioning them from dbprovs 
[09:51:03] <jynus>	 I got now the codfw warning: Last snapshot for x1 at codfw (db2197) taken on 2026-08-05 23:00:06 is 386 GiB, but the previous one was 434 GiB, a change of -11.0 %
[10:20:06] <cezmunsta>	 marostegui: the ops.wmf_slave_wikiuser_sleep event ... it seems as though it can produce errors (unkown thread_id when attempting to kill by the look of it) in the logs, is that common?
[10:23:31] <marostegui>	 yeah, that typically happens when there's a race condition on attempting to kill a connection that is already gone. That's common
[10:29:09] <cezmunsta>	 Wouldn't it make sense to handle that?
[10:30:05] <marostegui>	 I don't think it is causing lots of issues
[10:30:38] <marostegui>	 We can create a task with low priority if you'd like, but I don't think it is something we have to address quickly
[10:32:27] <cezmunsta>	 Sure, it runs every 30s anyway and the gaps in the logs are much longer than that. I was just thinking about if there are queries that should be killed after that disconnected one causes the abort. 
[10:33:21] <cezmunsta>	 I'll add it in status "refine" for now 
[10:33:36] <marostegui>	 sure, and make sure to state it as low priority
[10:33:53] <marostegui>	 and that one kills only sleep ones anyway
[14:30:14] <Emperor>	 https://phabricator.wikimedia.org/P95935 <-- anyone care to tell me how to make "swift stat" tell me about this object?
[14:33:15] <Emperor>	 Ah, got it
[14:33:30] <Emperor>	 printf -v x "1/11/Flag_of_Sri_Lanka.svg/22px-\r\n      Flag_of_Sri_Lanka.svg.png"
[14:33:43] <Emperor>	 swift stat wikipedia-commons-local-thumb.11 "$x"
[14:34:06] <Emperor>	 (there are 6 spaces between \n and F)
[14:34:40] <Emperor>	 I'm pretty sure we should delete that, and not just because it's annoying and surely impossible to actually have in a wiki page...
[14:34:54] <Emperor>	 also it's from 2017, so I think escaped Amir1's deletion of thumbs from that container
[16:19:01] <Amir1>	 I think my script doesn't understand files with \n in them :( 
[16:19:58] <Amir1>	 everything else is fine