[01:08:42] PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [01:15:56] RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [09:19:09] @Amir1 I'm on a meeting with arnaudb, will let him go soon :-D [09:19:20] sure no rush [10:44:36] jynus: just a warning, I tagged you on T345521 and the image concerned is of naked genitalia [10:44:37] T345521: Original version of File:2008 scalpelless vasectomy, post-op.JPG has disappeared - https://phabricator.wikimedia.org/T345521 [12:06:55] Emperor: sorry, was fixing other stuff. I will do archeology of metadata and backups (I actually store data from pervious versions, too), but the ticket, at first seems weird [12:07:33] It looks a regularly deleted file, and with that, it is not something we handle [12:09:44] it may be some kind of metadata issue, once things get deleted and moved [12:12:19] I'm a bit confused about how renames work, to be honest :) [12:12:25] bad [12:12:57] ah, c'mon, it's not that terrible a personal failing :) [12:13:05] [I know what you meant] [12:13:18] I was going to start sending you tickets your way :-D [12:15:14] one thing I sure is that image didn't exist with that name in the last 2 years, otherwise it would have been backed up [12:15:49] but it may have been under other name, I will search its hash and find it [12:16:13] and most likely it won't be lost from swift, just its metadata wrong [12:23:49] I am now going to check if the file has been hard deleted or metadata changed by checking old backups [12:24:56] in terms of data we have it, we can recover it, the issue is backup metadata also don't have a record of that so I'm afraid the loss happened long time ago, unless db backups deny it [12:32:26] AFAICT in this particular case we still have a deleted object that resulted from the original rename event [12:37:36] I am not so sure it was renamed, see how in 2015 it's title was used as is now: https://de.wiktionary.org/w/index.php?title=Ei&diff=prev&oldid=4586041 [12:39:00] and indeed 3 months ago, the image existed on the image table [12:40:58] oh, I see- I think there is logic on backups, that if a file disappears but the same file with the same sha1 is with another title, we assume it is rename [12:41:12] so it will be on the backup archive metadata [12:41:50] so the good news is that both metadata and file can be recovered no problem [12:42:02] what's missing is knowing why this occurred [12:44:52] sadly uploading a newer version of the file makes things more complicated [12:45:30] yeah, that definitely didn't help [12:46:55] so oi_archive_name would normally contain the name of the file where the object was moved when uploading the new version [12:47:24] it is blank as I guess the rename failed, as it didn't exist [12:47:43] but could I ask you to find it if I give you some parameters? [12:49:09] it would be on wikipedia-commons-local-public.10 container [12:50:57] and then /archive/20110424153959! but the number may change a few seconds up or down [12:51:42] don't remember if it uses 1/10/ or not for the archive [12:55:01] it does contain things like archive/1/10/ ... but not archive/1/10/20110424153959 [12:55:39] moving upwards as it were, the first thing I get is archive/1/10/20110424230221!Sclerites_of_insect_wing.svg [12:56:04] ( from swift list wikipedia-commons-local-public.10 --prefix archive/1/10/20110424 ) [12:56:26] so we confirm that when todays file was uploaded, the file was not present [12:56:52] on my records I confirm (the restore-files is a bit confusing) that I got 2 records of backups [12:57:11] 2008_* and DSC_* [12:57:12] confirm> because the archive would contain the upload_date ? [12:57:23] archive filename, I mean [12:57:41] yeah, the metadata didn't find the file to rename [12:58:00] OK [if you could put that onto the ticket that'd be super :) ] [12:58:00] so it was not a case of references lost [12:58:31] also I didn't backup the 2008 file because I found the replica before, but the metadata was there [12:58:48] but I think I download it anway at the time [12:59:01] to make sure the sha1 was correct and to calculate the sha256 [12:59:13] so a few years ago, the file was there [12:59:36] now, the question is how to go about this- which is why we haven't automated this [12:59:57] should we just upload a new version with the original file, or should we try to make mw consistent? [13:00:45] I think the first will be easier, so I will attach the file to the ticket, and wiki admins can handle [13:02:03] I think that's likely to be simplest [13:02:32] I wasn't sure about who should do that sort of thing (having found the deleted object) [13:03:32] then thing that is scary is that there is not reason why that happened- there is no record touching that file [13:03:48] I will summary my findings on ticket [13:04:18] I'm actually quite happy because my backup system is working for the first time, we build it for this! [13:12:43] wait, one second, Emperor [13:13:12] try the same location, but the seconds should be the one of the new upload [13:13:58] so 20230904032905 or somesuch? [13:14:00] so 20230904 [13:14:04] yes, please [13:14:21] shouldn't be there many on the same day for that container [13:14:39] swift list wikipedia-commons-local-public.10 --prefix archive/1/10/20230904 -> archive/1/10/20230904042305!Richard_M._Wynne.jpg [13:14:40] my guess it still won't be there [13:14:52] but I said it wrong before [13:15:32] you were right this time though [13:15:43] and previous day, just to be thorough= [13:15:56] 20230903 [13:17:00] that gets 4 hits, but none of them obviously relate to vasectomy [13:17:13] ok, thanks [13:17:39] I expected that, but wanted to be sure 100% as the name is based on the time of the new upload, not the original upload [13:17:56] sure [13:21:31] I will also make the recovery code less ambiguous so I do a left join rather than a join [13:23:49] does anyone know the syntax to link, but not to show thumbnail on phab? it is for a friend [13:25:45] not guilty [13:26:05] I think it is just the name FXXXXX [14:21:41] s2 and x1 snapshots are running now, should fix ongoing issues