[06:56:53] <Amir1>	 sigh, I woke up a bit late, Doing topology changes now
[07:21:38] <marostegui>	 what's wrong with clouddb1021? any task where I can read?
[08:11:42] <Amir1>	 marostegui: what is wrong about it?
[08:12:03] <marostegui>	 it is not accessible from cumin
[08:12:56] <marostegui>	 I will create a task
[08:13:30] <marostegui>	 actually any of them are anymore
[08:13:55] <marostegui>	 it is related to ipv6 I think
[08:14:16] <Amir1>	 yeah, I can't connect to it either
[08:16:28] <Amir1>	 marostegui: okay if I schedule switchover tickets for the next week?
[08:16:32] <Amir1>	 s1 and s3
[08:16:38] <marostegui>	 sure
[08:16:43] <Amir1>	 Awesome. Thanks
[08:16:49] <marostegui>	 I have created https://phabricator.wikimedia.org/T323550
[08:19:34] <taavi>	 the issue is that mysql_root_clients (which is used for the firewall rules) is for ipv4 only, so you can't connect via ipv6
[08:19:46] <taavi>	 but a better question is why do those hosts suddenly have ipv6 records
[08:20:27] <marostegui>	 taavi: Thanks for the insights. Yeah, I don't know what could have changed lately (as I have been away pretty much since Wed past week)
[08:21:04] <taavi>	 I note that maintain-dbusers is also failing because the grants are for ipv4 only
[08:21:51] <marostegui>	 taavi: Yeah, we don't have grants for ipv6 https://phabricator.wikimedia.org/T270101
[08:28:00] <Amir1>	 https://wikis.world/@nixCraft@mastodon.social/109385543340690248 xD
[08:30:12] <Emperor>	 :)
[08:30:19] <Emperor>	 6a) restore costs extra
[08:49:09] <moritzm>	 quick reminder: I'm going to reboot cumin2002 at 9:00 UTC, i.e. in ten minutes
[09:10:20] <jynus>	 moritzm: let me know when finished
[09:15:31] <moritzm>	 jynus: reboot is done, the only thing missing is to rearm the keyholder for homer, which should also be done soon
[09:15:40] <moritzm>	 but cookbooks etc can resume
[09:27:14] <jynus>	 thanks
[10:28:39] <jynus>	 I am going to test a new version of transfer.py on cumin2002, please don't use it for any transfer in the following 2 hours or so
[10:32:18] <marostegui>	 jynus: can we use it on 1001 though?=
[10:32:23] <jynus>	 yep
[10:32:32] <jynus>	 technically you can use it on cumin2002 too
[10:32:40] <jynus>	 but I cannot guarantee yet it will work
[10:32:58] <jynus>	 I am checking I haven't broken compatibility and options
[10:33:13] <marostegui>	 no worries, will stick to 1001
[10:36:56] <jynus>	 I will got you a manpage too, what is this, the 80s!??
[10:39:07] <marostegui>	 did you upload transfer.py to freshmeat.net too?
[10:40:13] <jynus>	 I don't get that reference, was that like softonic?
[10:41:35] <marostegui>	 https://www.rigacci.org/docs/biblio/online/debian_survival/freshmeat-page.png
[10:41:38] <marostegui>	 don't remember this site?
[10:42:16] <jynus>	 that was part of the conglomerate of slashdot, maybe?
[10:42:25] <marostegui>	 I think sourceforge owned it
[10:42:28] <marostegui>	 but I don't remember it
[10:51:46] <jynus>	 in any case, this is the highlights on the changelog: https://gerrit.wikimedia.org/r/c/operations/software/transferpy/+/859455/2/debian/changelog
[11:16:48] <jynus>	 marostegui: if you were attached to your cumin1001 screen session, I may have detached accidentally (screen itself wasn't affected)
[11:16:55] <jynus>	 *detached you
[11:17:08] <marostegui>	 jynus: nope, you didn't apparently 
[11:17:09] <jynus>	 I got confused between my and your screen session, that's all
[11:17:46] <jynus>	 and forced the -d later to realize I was on the wrong session
[12:59:34] <jynus>	 marostegui: I am now confident enough to recommend using transfer.py on cumin2002
[13:00:09] <jynus>	 I will leave it like that for a day so it has a full rotation of backups using it
[13:00:18] <jynus>	 and then upgrade cumin1001 too
[13:00:40] <marostegui>	 cool jynus 
[13:01:23] <jynus>	 it should not change anything for you, but you know how it is - it has so many dependencies that it is difficult to test every running part
[14:18:46] <moritzm>	 would tomorrow morning be okay to reboot cumin1001 from a backup and DB perspective?
[14:19:15] <marostegui>	 moritzm: From a DB point of view, maybe, I will only be able to tell you tomorrow morning :)
[14:26:06] <moritzm>	 okay, given that most people tend to use cumin1001 over 2002 and if it's fine for Jaime I'd announce something like 10 UTC to the ops mailing list? Then we have time in the morning to figure out whether we can proceed or not?
[14:30:11] <jynus>	 let me give you an approximate time on when backups finish
[14:30:30] <moritzm>	 ack
[14:31:36] <marostegui>	 moritzm: I think 10 UTC should be fine for me, but I will be able to tell you tomorrow morning 
[14:32:06] <jynus>	 6h30 - 19h should be backup-free on eqiad
[14:34:16] <moritzm>	 ok, thanks. I'll send a mail announcing a reboot for tomorrow 10 UTC; and if that needs to be moved, so be it
[14:36:07] <Amir1>	 I always love dropping stuff https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=db1122&var-datasource=thanos&var-cluster=wmcs&from=1669084515584&to=1669127715584&viewPanel=28
[14:48:32] <Amir1>	 moritzm: I have seven screens in cumin1001 right now :P 
[14:48:39] <Amir1>	 but it should be done by tomorrow
[14:49:01] <moritzm>	 ack :-)
[16:38:33] <Amir1>	 hmm, it's the combination of doing alter on externallinks plus dropping a field with forty bytes from index. Commons is 70GB smaller because of that change only https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=db2119&var-datasource=thanos&var-cluster=wmcs&from=1667972777502&to=1668222126671&viewPanel=28
[16:38:40] <Amir1>	 the rest is still to be done