[07:34:12] <marostegui>	 I have restored pc1012 back as pc2 master
[07:36:03] <Amir1>	 Thanks! and also thanks for the help in switchover of s4
[07:45:29] <marostegui>	 Amir1: on db2094:3318:                 Last_SQL_Error: Column 2 of table 'wikidatawiki.templatelinks' cannot be converted from type 'bigint' to type 'varbinary(255)'
[07:46:01] <Amir1>	 ugh
[07:46:04] <Amir1>	 let me fix it
[07:46:19] <marostegui>	 I haven't checked what is going on, as I am busy with some other stuff
[07:47:04] <Amir1>	 marostegui: don't worry, I got this one 
[07:47:16] <Amir1>	 let me do some work from time to time, you're making me lazy
[07:49:45] <Amir1>	 replication flowing now
[07:50:23] <Amir1>	 this is combination of a couple of issues, mostly caused by changes to auto schema to make it work with multidc, I'll make a patch for each one
[07:51:39] <marostegui>	 good, thanks!
[07:56:10] <marostegui>	 Amir1: you running the schema change for https://phabricator.wikimedia.org/T312160 ?
[07:58:22] <Amir1>	 marostegui: si
[07:58:43] <marostegui>	 cool, remember that it cannot be done on the master
[07:58:47] <marostegui>	 it needs a switchover
[07:59:01] <Amir1>	 yup, I leave that part to you mwhaha
[07:59:21] <Amir1>	 note that we now we need to do switchovers on codfw and eqiad 
[08:00:30] <marostegui>	 what do you mean?
[08:00:39] <marostegui>	 x1 codfw master isn't used
[08:00:42] <marostegui>	 it can be run there
[08:02:10] <Amir1>	 I think mediawiki.org is already multidc
[08:02:28] <Amir1>	 and if you hit mw.org, you hit x1 (echo, etc.)
[08:03:09] <Amir1>	 if mw.org is not multidc, it'll be soon so we should hurry :D
[08:03:19] <marostegui>	 oh, I thought only x2 would be affected by that
[08:03:20] <marostegui>	 :(
[08:03:56] <jynus>	 marostegui: sorry to interrupt conversation, m1 has no blocker, I can proceed to shutdown bacula, right?
[08:04:09] <Amir1>	 nope, if you hit a page in basically any wiki, you get db reads on section of that wiki + s8 + s4 + s7 + x1 + x2 + pcs
[08:04:14] <marostegui>	 jynus: yeah, anytime you want, the switchover is scheduled for .30 though
[08:04:22] <marostegui>	 Amir1: :)
[08:04:23] <marostegui>	 ok
[08:04:28] <marostegui>	 that is good to know
[08:06:18] <jynus>	 it is ok, as it has a lot of different alerts, I can start downtiming early
[08:06:48] <Amir1>	 btw, I'm almost done with migrating replication lag alert to alert manager (off icinga) heads up
[08:07:08] <Amir1>	 probably will be done today
[08:07:12] <marostegui>	 see _security
[08:07:14] <marostegui>	 jesus christ
[08:07:21] <marostegui>	 we were almost done :(
[08:10:04] <Amir1>	 at least I have an excuse to automate this
[08:10:11] <Amir1>	 *now I have
[08:17:40] <jynus>	 I don't remember much what was the procedure for non-backup services of m1, etherpad required a restart? or killing old connections was enough?
[08:17:56] <marostegui>	 It depends, sometimes it does
[08:18:01] <marostegui>	 Some others it doesn't
[08:19:34] <jynus>	 ah, it is actually nicely documented: "Normally systemd takes care of it and restarts it instantly. However if the maintenance window takes long enough, systemd will back out and stop trying to restart, in which case a systemctl restart etherpad-lite will be required. "
[08:29:49] <marostegui>	 I am going to start m1 failover
[08:31:41] <marostegui>	 etherpad seems to be working without a restart
[08:31:45] <jynus>	 same
[08:32:13] <jynus>	 will wait for other services to confirm they are right before restarting backup ones (I will also restart them too)
[08:33:05] <marostegui>	 everything seems fine
[08:33:08] <marostegui>	 librenms, rt..
[08:33:36] <marostegui>	 Can I merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/826223 ?
[08:33:41] <jynus>	 I was about to ask
[08:33:45] <jynus>	 +1 from me
[08:34:16] <marostegui>	 merged
[08:34:39] <jynus>	 will run the failed x1 backup when I am done with bacula for testing purposes
[08:53:01] <jynus>	 I belive m1 performance could have been sligthly affected due to the reboot - nothing serious or blocker, my guess is it will take a few minutes to get to the previous level
[08:53:18] <marostegui>	 yeah, probably
[08:54:16] <jynus>	 as some bacula monitoring process ran slower than usual
[08:56:33] <marostegui>	 Can I get a quick IPs review https://gerrit.wikimedia.org/r/c/operations/puppet/+/826506 
[08:56:34] <marostegui>	 ?
[08:59:56] <marostegui>	 thanks!
[09:01:37] <jynus>	 so I have a few pending things regarding m1: re-enable and force rung "Content database backups remote long term backups" (Es-rw) and rerun x1 codfw snapshots
[09:02:44] <marostegui>	 anything I can do to help?
[09:03:15] <jynus>	 not really, just confirming there is no expected further maintenance on m1?
[09:03:35] <jynus>	 as es backups take many hours to complete
[09:03:51] <marostegui>	 nope
[09:03:52] <marostegui>	 nothing
[09:05:15] <marostegui>	 jynus: regarding m2 in eqiad, can I stop db1117:3322 to clone another host?
[09:05:23] <jynus>	 one last question, for any of you, I belive you mentioned ongoing x1 maintenance on codfw?
[09:05:41] <jynus>	 marostegui: yeah, I will only intend to work with x1 today
[09:05:53] <marostegui>	 jynus: I am not doing anything on x1 today or tomorrow no
[09:06:31] <jynus>	 I am asking because those are not in a hurry (unlike es), and maintenance would explain why thye failed a couple of times
[09:06:53] <jynus>	 so I can delay x1 if there is ongoing alters of heavy writing no issue
[09:07:01] <marostegui>	 there are big alters going on 
[09:07:03] <marostegui>	 but the hosts won't be down
[09:07:08] <jynus>	 ah, I see
[09:07:32] <jynus>	 xtrabackups is a bit sensitive to that kind of load, even if in theory it should work
[09:07:40] <marostegui>	 then yeah, it might take long
[09:07:44] <marostegui>	 but Amir1 is the one doing that one
[09:07:48] <marostegui>	 so maybe he can estimate better
[09:07:59] <jynus>	 which is a good thing ,because it means there is no programming issue elsewhere
[09:08:10] * Amir1 reads up
[09:09:00] <Amir1>	 yeah, it's 93GB, I think it'll take around ten hours for each replica
[09:09:12] <jynus>	 so is that done serially?
[09:09:29] <jynus>	 that way another thing I can do is swith to another replica
[09:09:45] <jynus>	 that has either already completed or not touched yet
[09:10:16] <jynus>	 or just wait, eqiad backup is ok, and it is only a couple of days old, not a big deal
[09:12:10] <Amir1>	 yeah, all schema changes are done serially 
[09:12:37] <jynus>	 I think it will just be easier to just wait, just ping me -if you can- when it finishes on db2101 (the backup source)
[09:14:18] <jynus>	 schema changes > backups being perfectly up to date at all times
[09:15:02] <jynus>	 and if I am complaining because a redundant backup is "1 day, 6 hours old" we are in a great place :-D
[09:47:03] <marostegui>	 I have rebooted all standby proxies + codfw proxies
[12:02:16] <jinxer-wm>	 (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db1194:9104)  - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed
[12:02:49] <marostegui>	 ^ me
[12:07:16] <jinxer-wm>	 (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db1194:9104)  - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed
[12:19:25] <Amir1>	 jynus: The script I'm writing for reboot doesn't really distinguish between a backup source or another replica. I wonder how I can make sure it doesn't try to reboot a backup source while it's actually working. Can you give me an idea on this?
[12:20:04] <jynus>	 do you want to get the distinction or prevent from restarting while the dump is ongoing?
[12:20:20] <Amir1>	 yup
[12:20:24] <Amir1>	 the latter
[12:20:38] <jynus>	 if the latter, I already told Stevie Beth how to monitor for ongoing backups- a dump user is connected
[12:21:07] <Amir1>	 ah, thanks
[12:21:20] <jynus>	 but I would like to handle backups at my own pace, to be fair- I would use your script, but I would prefer to control when to restart them myself
[12:22:11] <jynus>	 let me think, because that would work for es hosts, but not sure if for snapshots
[12:22:30] <jynus>	 as I belive a root user is used for xtrabackup
[12:22:51] <jynus>	 (as it doesn't use mysql except for checking replication status, the copy is of the filesystem)
[12:23:11] <jynus>	 for that, probably you will want to check for a lock file set by transfer.py
[12:25:29] <jynus>	 as for how to distinguish, either with puppet role (mariadb::backup_source) or zarcillo
[12:27:11] <RhinosF1>	 is kormat ok? they haven't been active (anywhere i can see) in a while.
[12:28:08] <jynus>	 pmed you
[12:28:53] <Amir1>	 yes, please let's not discuss people's private lives in publicly logged channels
[12:35:28] <jynus>	 Amir1: so there will be a specific lock dir under /tmp for snapshots
[12:35:42] <jynus>	 I am trying to remember the name
[12:35:56] <jynus>	 it exists only during transfer
[12:36:01] <Amir1>	 jynus: is that for ES or core? For core I assume the dump user would be enough
[12:36:05] <jynus>	 yeah
[12:36:07] <Amir1>	 I'm not planning to run it on ES
[12:36:20] <jynus>	 so the dump user is for dumps
[12:36:36] <jynus>	 and the /tmp dubdir is for snapshots, only on backup sources
[12:37:24] <jynus>	 the first should probably be done for other users (webrequest user, admin user, so shouldn't be a huge overhead)
[12:37:50] <jynus>	 I'm trying to find you the docs or source code for the second
[12:38:08] <Amir1>	 if there is any backups running, I can go and poke around :D
[12:38:28] <jynus>	 mmm, let me see if x1 finished
[12:38:52] <jynus>	 I gave it a try expecting to fail
[12:40:15] <jynus>	 actually it finished, but correctly
[12:40:23] <jynus>	 do you want me to run one somewhere?
[12:41:40] <Amir1>	 nah, let's wait
[12:41:42] <Amir1>	 this is not urgent
[12:41:50] <jynus>	 it is ok, I am curious myself
[12:41:50] <Amir1>	 just ping me if something starts
[12:41:56] <jynus>	 it is just one command
[12:42:00] <Amir1>	 haha sure
[12:42:04] <jynus>	 actually I want to show you how to run it
[12:42:09] <jynus>	 go to cumin2002
[12:42:22] <jynus>	 start a screen session (or tmux)
[12:42:43] <jynus>	 and run "remote-backup-mariadb s5"
[12:43:14] <jynus>	 (it should be super easy to start a backup!)
[12:43:34] <Amir1>	 sure
[12:44:01] <Amir1>	 started
[12:45:04] <Amir1>	 hmm, I don't see replication stopped in backup sources of s5
[12:45:05] <jynus>	 that should be running on db2101.codfw.wmnet
[12:45:39] <jynus>	 oh, it depends on configuration, only stop it based on load- see config at /etc/wmfbackups/remote_backups.cnf
[12:45:48] <Amir1>	 I see
[12:45:53] <jynus>	 I think I stop it on s1, s8 and s3
[12:46:11] <Amir1>	 show processlist doesn't bring dump user from what I'm seeing
[12:46:14] <jynus>	 because otherwise it writes too much and fails more often (plus make it faster)
[12:46:26] <jynus>	 yeah, that is because it is a snapshot, it copies from the filesystem
[12:47:10] <jynus>	 but there is /tmp/trnsfr_dbprov2003.codfw.wmnet_4400 on the source
[12:47:58] <jynus>	 if a /tmp/trnsfr_* dir exists, it is transferring (that way I lock against 2 processes running at the same time on the same port)
[12:48:24] <jynus>	 the rest is where it is being sent and through which port, but you don't care about that :-D
[12:48:52] <jynus>	 4400 is the default, but it will use more 4401, 4402 if the others are busy
[12:49:17] <jynus>	 so checking the existence of /tmp/trnsfr_* should be enough
[12:49:36] <jynus>	 and for dumps, the dump user 
[12:51:57] <jynus>	 and note a transfer or dump can take multiple hours, es dumps now take 27+ plus due to its low concurrency
[12:53:40] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s6 on db2114 is CRITICAL: 542 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2114&var-port=9104
[13:05:21] <Amir1>	 That was me, it should be back now
[13:08:44] <Amir1>	 yeah, if see it, I just skip it
[13:12:05] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s6 on db2114 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2114&var-port=9104
[13:17:41] <Amir1>	 jynus: SELECT * FROM information_schema.processlist WHERE User like '%dump%' does this look good to you?
[13:18:03] <jynus>	 we can simulate this too, let me see
[13:18:29] <jynus>	 note we prefer sys.processlist as it is non blocking
[13:18:51] <jynus>	 but should be ok if not used under an outage
[13:19:45] <jynus>	 but yeah, that should be ok --> if it returns rows, wait/abort
[13:20:14] <jynus>	 try to make the user configurable, you may know why :-D
[13:24:06] <jynus>	 another potential way to detect snapshots would be "pgrep xtrabackup"
[16:00:47] <jynus>	 I think I can finally run the delayed es long term backups