[06:29:10] I'm going to start with m2 switch [08:21:37] taavi: https://gerrit.wikimedia.org/r/c/operations/puppet/+/977123 [08:24:53] marostegui: +1'd. also, just to make sure you're aware, the table is also there on private.dblist and fishbowl.dblist wikis, not just on centralauth [08:31:21] taavi: Oh, I wasn't aware, thank you [08:32:02] Just made a comment on the task so I don't forget [09:24:27] I am going to test database backup tests new package on backupmon1001 (alerts will be disabled for a while there) [09:50:46] marostegui: a major change for x1 binlog just got merged, shall we let it ride with this week's train or it's too urgent and I need to backport it? [10:01:36] We can wait I think [10:02:16] let me know if things get worse and we need to do change something [10:03:11] Let me clean binlogs now just to be fully sure [10:07:35] done [10:07:45] we should be good until the patch is merged [10:20:49] awesome. thanks! [11:09:26] thoughts from non-backup people: is "backupcheck" a clear enough user name? Too specific? Too generic? https://gerrit.wikimedia.org/r/c/operations/puppet/+/977603 [11:13:15] is the database user also "backupcheck"? [11:14:28] (and what sort of privileges does it have?) [11:16:23] Emperor: yes, as the commits says [11:16:33] it has SELECT on the dbbackups db [11:22:40] https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/profile/templates/mariadb/grants/production-m1.sql.erb;593519ff3e4de4ef707223ca0a9f5646886b3546$143?as=source&blame=off [11:26:48] seems like a reasonable name to me, then. [11:27:19] (alternatively, call it MOSS ;p) [11:28:17] there is one thing that I don't like, which is it creates one sudoers line per parameter combination (we have ~60 checks) [11:28:47] but given 1) this way it is fully automated 2) it avoids potential injections in my script, I will leave it like that [11:37:05] [I generally hate sudo, but this is not the time for that argument :) ] [12:40:04] Emperor: I can see why: https://gerrit.wikimedia.org/r/c/operations/puppet/+/977603/comments/b8d512ac_102df5ba [12:43:54] :-/ [12:45:13] "do you want to execute a command with a file with spaces? Too bad" [12:45:33] (I understand it was not built for that, but still) [14:00:12] spent a little time over the weekend on T350917 and again this morning, but my conclusion is it's an upstream-of-swift issue (and, particularly, not related to the nginx->envoy move) [14:00:13] T350917: Incomplete files uploaded (10 MB interruption) - https://phabricator.wikimedia.org/T350917 [14:30:25] got distracted, connecting in 1 minute [15:35:27] jbond: backupmon1001 has been updates, no blocker for puppet 7 anymore from my side [16:14:14] jynus: cheers [17:56:20] jynus: when is the next run for db1217 (misc m3) backups? [17:56:35] So I know if I can stop it between now and the next backup run [17:56:38] they all run tonight [17:56:48] Awesome, so tomorrow morning should be fine [17:56:55] Let me check, I think they start at 0h [17:56:56] I'll check it though to make sure its finished [17:57:01] but m3 I think is otrs [17:57:09] no [17:57:10] m3 is phab [17:57:10] which means takes a lot [17:57:12] ah [17:57:15] m2 is otrs [17:57:17] then it should be finish soon [17:57:21] let me give you a guess [17:58:08] should be finished at 6am UTC, at the very late [17:58:27] great [17:58:30] or you can check at http://localhost:8000/dbbackups/jobs/?search=dump+m3 when you get up [17:58:31] I will double check though [17:58:44] Yeah, I usually check in m1 backups table [17:59:01] or here: http://localhost:8000/dbbackups/eqiad/m3/dump/ [17:59:36] do you have the tunnel command handy? [17:59:38] I can also generate you one [17:59:47] yep, let me give you the web [17:59:57] oh is it on wikitech? [17:59:59] I can generate you one custom when you want if that helpe [18:00:01] d [18:00:24] https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Dashboard [18:00:35] thanks :* [18:01:00] or you can keep using the db, they are the same data :-D [18:01:19] yeah I usually use the table for historic reasons! [18:03:25] note also breaking this backups is not a big deal- it was only es before what was a pain (2 day backup)