[09:28:18] hi folks, could I get a +1 on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1264368 please, to drain the next 3 codfw ms backends? [10:05:55] Emperor: {{done}} [10:06:14] TY <3 [13:38:03] @Amir1 @marostegui I spoke with Moritz about nftables and opened https://phabricator.wikimedia.org/T421705 [14:03:58] I just noticed one thing, circular replication seems to be enabled on es7 [14:04:12] mentioning in case it was not intended [14:04:19] https://orchestrator.wikimedia.org/web/cluster/alias/es7 [14:04:48] vs https://orchestrator.wikimedia.org/web/cluster/alias/es6 [14:05:18] not a biggie, but it seems weird one it is setup and not the other [14:05:23] *one is [14:06:28] odd, looking [14:07:10] like, I can see it being on on those, like for pc/x* as there should be no schema changes, but either the 2 should be on or 2 off [14:07:26] maybe a script issue? [14:08:21] likely [14:10:26] jynus: yep: https://phabricator.wikimedia.org/T416708#11764723 [14:13:10] federico3: ^ [14:13:23] yes, I'm reviewing the logs [14:16:09] indeed it's missing, I can run it now [14:22:07] why was it skipped, was it done manually ? [14:22:50] I am guessing it was done per section and skipped [14:24:13] yes, we don't have an automated process, I just started collecting documentation days ago in https://wikitech.wikimedia.org/wiki/MariaDB/Datacenter_switchover [14:24:24] we have an automated process [14:24:36] riccardo did it [14:25:12] ah, do you have a link to it? [14:25:44] sre.switchdc.databases.prepare [14:26:27] and I guess sre.switchdc.databases.finalize [14:27:02] afaik that's for the setup part (mentioned in the wiki page) [14:27:15] I mean, one cannot use the manual section selection and then claim "it is not automated" [14:27:17] yes, they do changes in individual sections but no monitoring [14:27:33] it does monitoring, it checks after it is done that it worked [14:29:42] you mean it should be run without --section and without need to review the dashboard? That would make the process simpler [14:30:04] it is ok to do it manually if one wants to test it [14:30:46] but it is like running something with --ask-for-confirmation and then complaining it asks for confirmation :-D [14:30:49] don't you think? [14:32:50] the example commands in https://phabricator.wikimedia.org/T416706 included the section so I used the section; if we can use it without I'm happy to update the documentation to let it run automatically [14:37:21] see here: https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/switchdc/databases/__init__.py#L31 [14:38:16] The option is there, whether one wants to do it manually or automatically, the automation is there; it would be a decision you (dbas) take, not a lack of automation 0:-) [14:41:15] and this is where they are defined: https://github.com/wikimedia/operations-software-spicerack/blob/master/spicerack/mysql.py#L29