[04:30:32] <legoktm>	 hi, could someone look at https://phabricator.wikimedia.org/T315442 ?
[04:31:20] <legoktm>	 er no, sorry, Data-Engineering != Data-Persistence
[05:18:57] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 116 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[05:20:49] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 61 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[05:22:07] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[05:22:25] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[14:20:49] <Emperor>	 sadness, lots of objects that appear in container listings but then say 404 if you try and look at them
[15:25:59] <jynus>	 as in, real live queries to swift?
[16:16:50] <Emperor>	 jynus: I've tripped over them by running rclone and investigating its errors
[16:17:19] <Emperor>	 I don't _think_ these are things currently being served IYSWIM
[16:17:20] <jynus>	 remember I have a copy of originals, if you belive they could be lost
[16:17:25] <jynus>	 I see
[16:17:37] <Emperor>	 but I don't know how one would check more accurately (or in bulk)
[16:17:58] <jynus>	 (I have some tools precisely for checking :-D)
[16:17:59] <Emperor>	 jynus: oh, while you're here, T326965 was tagged sre-swift-storage which I think is wrong, is there a backups-specific tag I should apply instead?
[16:17:59] <stashbot>	 T326965: Q4:rack/setup/install backup2010, backup2011 - https://phabricator.wikimedia.org/T326965
[16:18:29] <jynus>	 yes, don't worry, I will take care (I didn't know they were created by robh)
[16:19:48] <Emperor>	 TY
[22:51:49] <Emperor>	 27,143 objects in listings but not actually there.
[22:52:16] <Emperor>	 Monday's problem
[22:52:31] <RhinosF1>	 Emperor: it’s 11pm on Friday. Why are you here
[23:00:29] <Emperor>	 I sleep better knowing the thing I left running has finished :)