[01:06:58] PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 3.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [01:07:09] PROBLEM - MariaDB sustained replica lag on m1 on db2078 is CRITICAL: 4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2078&var-port=13321 [01:08:12] RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [01:08:24] RECOVERY - MariaDB sustained replica lag on m1 on db2078 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2078&var-port=13321 [07:55:07] I just closed https://phabricator.wikimedia.org/T298585 [07:55:09] \o/ [07:59:33] nice [09:26:33] marostegui: let me know when/if you have time to test the querying and deletion of media backups files, I have updated the documentation: https://wikitech.wikimedia.org/wiki/Media_storage/Backups [09:27:46] jynus: sure! after lunch is ok? [09:28:01] I have itc at 15 CEST [09:28:07] so before or after [09:28:21] sure, I will read the doc anyways before :) [09:39:20] ping me before running as I only have the latest package installed on a single location [09:40:03] sure! [09:40:38] and thank you for this, other people helping me is super useful for things like this! [09:41:37] good morning [11:44:59] moritzm: the revision fixes is done now, the repool finishes in an hour so if you want to do it tomorrow or later today, it'd work for me [11:46:14] I'd aim for Monday morning if that's a fine time for backups/DBs (in the middle of the day is a pesky time for cumin reboots, too much activity) [11:48:06] WFM [11:49:33] me too [11:51:40] works for me [13:32:08] I'm sending a mail for Monday morning 7:30 in a bit [14:20:42] jynus: the doc is pretty clear, which name or file should I try with? (always dry run) [14:35:55] sorry, I was busy at a meeting [14:36:16] I know :) [14:38:08] could you try to find File:Tiff test.tiff from testwiki and dry-run delete it? [14:38:15] the latest package version [14:38:21] is on ms-backup1001 [14:39:18] sure [14:39:20] please help me giving me feedback on anything [14:39:31] ui, clarity of docs, clarity of cli messages, etc. [14:39:36] will do, let me follow the doc for that file [14:39:40] even if small, I can write it down [14:40:05] and give it lower preference, but the main question is "would this be clear/easy to follow for a deletion request?" [14:40:26] jynus: Yeah, I was going to suggest a section just for that [14:40:31] So: this is the long explantion [14:40:39] and this is what you want to do during an emergency [14:40:41] ssh blablabla [14:40:43] sudo delete... [14:40:46] well [14:40:59] backup deletions are not usally an emergency [14:41:04] ok then [14:41:11] but yes, anything you can provide for improvement [14:41:22] even if not super improtant, please say it [14:41:39] or you can send it to me in private if you prefer [14:43:30] thank you again [14:44:29] So ok done [14:44:41] It is pretty straightforward like the other script I mentioned during the meeting [14:44:56] anything weird unexpected/ errors? [14:45:05] anything you would like to have extra? [14:45:34] I would add a reminder to make sure it gets delete from the other DC (like the doc says) [14:45:42] ah, that is great feedback [14:46:08] what about help? What would be the easiest? --help, man page? [14:46:55] So there's one thing I find strange [14:46:59] tell me [14:47:08] By default it is dry-run right? [14:47:11] yes [14:47:23] for that I write a reminder [14:47:24] I would add that header that too once you start it [14:47:30] Yeah you did but after the whole process [14:47:35] # delete-media-file [14:47:35] Wiki for deletion [default: commonswiki]: [14:47:39] I would add it there too [14:47:39] so, at the start? [14:47:41] yeah [14:47:47] before any input? [14:47:51] cause if yeah [14:47:53] ok [14:47:57] Otherwise it can look scary [14:48:00] that's also helpful and super simple to add [14:48:58] e.g. "you are about to do an actual deletion, but will be asked for a final confirmation" when --execute [14:49:05] Yeah [14:49:12] and "This is a dry run bla bla bla" for a dry run [14:49:17] ? [14:49:23] And when doing it without the execute something like: This is a dry-run you won't delete anything even if it looks like so" [14:49:24] yeah [14:49:26] something like that jynus [14:49:35] cool, I will add that [14:49:42] something else? [14:49:52] did you see the query one? [14:49:58] it is even simpler [14:50:10] query-media-file [14:50:48] That's the one I use I think for the request when you were out? [14:50:51] Or not? [14:50:56] An no, I used the other one [14:51:00] no, you probably used the recovery one [14:51:05] yeah [14:51:08] but it is exactly the same, except that it ends [14:51:16] for maximum security [14:51:21] "I just want to query" [14:51:22] yeah [14:51:44] Maybe add that to the doc too? If you just want to query use -> query-media-file? [14:52:07] Technically it is here :-) https://wikitech.wikimedia.org/wiki/Media_storage/Backups#Querying_files [14:52:12] the first one [14:52:29] maybe either unclear or you thought it was the recovery one? [14:53:37] Yeah, I meant on the deleting files [14:53:40] just linking in case [14:53:45] but anyways, not big deal [14:53:57] on the doc or the output of the script? [14:54:07] on the doc [14:54:09] if it is just a link it is trivial to be added [14:54:12] thank you [14:54:39] "consider using X first bla bla bla" [14:56:35] Yeah something like that [14:56:42] https://wikitech.wikimedia.org/w/index.php?title=Media_storage%2FBackups&type=revision&diff=1993387&oldid=1993373 [14:56:44] Feel free to discard my feedback if it doesn't make sense! [14:56:51] it is all ok [14:56:58] That looks good to me [14:57:09] Btw I did this https://wikitech.wikimedia.org/w/index.php?title=Media_storage%2FBackups&type=revision&diff=1993373&oldid=1993284 [14:57:23] yes, thank you [14:59:53] anything else? The logs are a bit unreadable [15:00:07] but not sure how to format them better or make the messages clearer [15:00:30] everything you do should be on /var/log/mediabackups too [15:00:53] so later we can audit them, in case something wrong happens [15:00:59] indeed it is there [15:01:26] not too worried about logs, we can always improve them later, but feel free to provide feedback on that too [15:02:34] anything else? I don't want to steal a lot of your time? [15:03:47] I think that's all for now! [15:03:55] because how bad mediawiki metadata is, we will likely have edge cases I haven't taken into account [15:03:55] Maybe get someone less familiar with backups to run it [15:04:02] yeah [15:04:44] but appreciate your general aproval a lot [15:05:11] haha my pleasure! [15:05:33] I think I can take care about the extra message right away [15:05:41] \o/ [15:06:11] but yes, I think there should probably a procedure for deletion, separate from this manual [15:06:28] more about the context how to attend requests, etc [15:06:43] but not worrying about that at the moment [15:06:50] thank you again! [15:06:57] you are welcome!