[05:56:53] s4 switchover starting in a few moments, all clear marostegui? [05:56:59] yep all good! [05:57:06] +1 [06:11:31] Done [07:13:25] marostegui: why is the sql_mode only set to IGNORE_BAD_TABLE_OPTIONS? [07:15:37] cezmunsta: I don't remember to be honest, it's been like that for _many_ years, not sure if jynus remembers why. I think it was like that even before I joined [07:15:55] Yes, 2014-08-28 [07:16:08] so even before jynus [07:17:50] * cezmunsta hopes that the application code set its own [07:46:41] cezmunsta: it has been a fight for a long time, something more strict breaks mediawiki, there is a ticket about it [07:47:24] https://phabricator.wikimedia.org/T108255 [07:49:16] See how that was filed after my proposal, mere month after I got onboarded [08:23:21] hello, who owns dbprov2006 and what needs to be done with it before its switch downtime? also db2205 will need some special care next week - https://phabricator.wikimedia.org/T430910 [08:24:44] I do [08:24:57] normally nothing, backups usually run at night [08:29:00] cool [08:29:19] XioNoX: creating the switchover ticket for db2205 [08:30:58] <3 [08:38:37] sadly, a network outage casued errors on dbprov1004 backups, so retrying that [09:09:50] I'm going to switchover s5 in codfw @marostegui @cezmunsta [09:12:25] cezmunsta: same for db2159, db2160, sessionstore2004, restbase2036 in 2 weeks - https://phabricator.wikimedia.org/T430918 [09:16:27] XioNoX: ack - I will create a ticket for db2159 - jynus: does anything need doing for db2160? [09:17:24] I don't own db2160, that's your call [09:19:08] Sorry, I saw a non-3306 port and presumed that it was a backup instance [09:20:26] you have many multiinstance hosts still: misc, sanitarium [09:20:55] cezmunsta: same for db2162, db2251, pc2022 in 2 weeks also - https://phabricator.wikimedia.org/T430922 [09:21:41] jynus: it is in dbprov2005's config [09:21:56] XioNoX: ack [09:22:10] cezmunsta: sure, but es hosts too, doesn't change anything- you decide, I cope [09:22:40] generating backups are not a prioritary service [09:23:17] and in most cases they will retry automatically [09:43:56] cezmunsta: a few more, this time for the maintenance in 3 weeks: pc2017, db2229, db2230, es2046 - https://phabricator.wikimedia.org/T430928 [09:44:35] XioNoX: ack [09:53:24] federico3: can i test https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1291952? [09:54:28] @marostegui yes, or I can test it on my side if you prefer [09:54:42] up to you, can you do it now? [09:55:13] yes: which host is to be decom? [09:55:19] es1033 [09:55:46] What backs the switchmaster tool? Internal Server Error [09:56:00] cezmunsta: if the candidate master is depooled the tool will fail [09:56:12] Amir1: ^ [09:56:23] cezmunsta: LAST ONE for codfw pod A/B, and probably a small summer break - apus-be2004, db2185 for https://phabricator.wikimedia.org/T430929 [09:56:33] marostegui: this was for s6 [09:56:37] XioNoX: ack [09:56:54] XioNoX: that host needs nothing else than a downtime [09:57:26] marostegui: it would be awesome if zarzillo could tell me that :) cc federico3 ;) ;) ;) [09:57:43] marostegui: and who owns apus-be2004 ? [09:57:44] cezmunsta: db2214 looks depooled which is the candidate master [09:57:48] XioNoX: Emperor [09:58:12] btw Empero.r is out till 5th [09:58:13] XioNoX: uh? [09:58:24] marostegui: ah, I see [09:59:08] federico3: let me know when you are running the depool, so I can merge puppet [09:59:15] federico3: https://zarcillo.wikimedia.org/api/v1/can_be_depooled/db2185 it would be nice if the API endpoint could tell me that the host doesn't need a depool [10:00:33] @marostegui can i depool it right now? [10:01:12] federico3: sorry, it is depooled already, just the decomm script needs to run [10:05:15] @marostegui looks good https://phabricator.wikimedia.org/P94692 ? [10:05:19] federico3: is there any reason that db2214 is still depooled? [10:05:45] federico3: yes [10:06:25] federico3: "Slowly repool the db2214 if ready" is marked as done on https://phabricator.wikimedia.org/T430814 [10:06:51] then it was depooled for updates, I'll check in abit [10:07:17] It is preventing creating switchover tickets [10:10:00] marostegui: I would like to reboot clouddb1017 to verify T427060, are you using it by any chance? [10:10:00] T427060: clouddb1017 getting stuck during shutdown - https://phabricator.wikimedia.org/T427060 [10:10:40] dhinus: I am not useing it no [10:11:41] thx [10:15:41] @marostegui so far so good, we got to https://phabricator.wikimedia.org/P94695 [10:15:54] ok, let me merge [10:18:00] federico3: done [10:18:51] @marostegui ok, the cookbook completed with https://phabricator.wikimedia.org/P94695 (want me to do those steps?) [10:19:08] federico3: I can do them, thanks! [10:26:29] hmm restarting prom mysql exporter on db2213 got stuck [10:31:15] Opinions on running this https://phabricator.wikimedia.org/T430531#12071646 ? [10:34:30] claime: I personally prefer if things are run from a mw script instead of logging into a sql prompt and doing it just like that [10:34:53] @cezmunsta db2214 should be good to go in 20 min [10:35:37] marostegui: That would require creating a new script given the existing one is global and would recalculate edits for all accounts [10:35:46] It may be worth it don't get me wrong [10:36:17] claime: You asked for opinions and that's mine :). I'd prefer a script [10:36:58] marostegui: Yeah, I understand :D [10:37:12] I'm just giving context [10:39:14] Yeah. I just don't personally like running things just like that without all the safeguards and tracking a script using mw framework could provide [10:40:14] ack [10:42:36] federico3: ack ty [11:59:56] I plan to reboot cumin2002 later the day and wanted to doublecheck that all scheduled DB and backup tasks are moved to cumin2003 at this point? [12:01:12] moritzm: I have none, I am in a meeting with federico3 and cezmunsta at the moment, they'll probably answer later [12:03:36] sure, no rush. thanks [12:24:43] marostegui: regarding that issue. The problem is that there is no other way to find replicas of a section without being in production and connecting to master. The only other way I can think of is to open the file of every db host in puppet to find the candidate master [13:02:43] heads up. I'm merging this [13:02:45] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1306797 [13:40:21] @marostegui @cezmunsta can I switchover s3 codfw master as well? [13:41:06] federico3: there is a ticket for that [13:41:15] https://phabricator.wikimedia.org/T430912 [13:42:05] .. plus the related task for the switch maintenance [13:42:17] are you planning to do it? [13:43:29] I was going to do it in time for the maintenance, but if it is done now then we just need to make sure that it isn't undone for the maintenance :) ... so you can take it if you like? [13:43:50] Is it for the reboots? [13:43:56] this is codfw so we can do the switchover now [13:43:57] yep [13:44:10] +1 [13:45:02] re codfw: I was meaning, doing it now does require a re-check at the time of maintenance to make sure that it is still a normal depool :) [13:46:08] I have reassigned it to you [13:46:09] yes the recheck is always good practice [13:46:13] ok, starting in now [14:49:16] @cezmunsta do you have OS reimaging or other stuff going on on pc*? [14:49:28] No [14:50:24] ok I'll do a rolling restart of pc sections [14:59:59] there's an alert "PROBLEM - MariaDB Replica IO: pc1 on pc2021 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master", is that expected due to the rolling restart? [15:01:46] yes but it should autosilence it, did it p.age? [15:03:05] @moritzm I'm not seeing it on AM [15:03:22] not a page, but it was logged to IRC in -operations [15:03:43] came here to check if that was some hardware failure [15:03:53] and then saw the note on the rolling restart [15:04:38] yup, it's the running restart [19:25:05] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db1228:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:06:39] Hey, I'm trying to clean up [[HAProxy]]. Is there any difference between the "HAProxy for MariaDB" and dbproxy? Are they one and the same? [21:07:04] If so, I'd like to move all that content in [[HAProxy]] over to [[Mariadb/dbproxy]] and then make [[HAProxy]] a disambiguation page [23:29:43] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db1228:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed