[06:09:34] Hi, I changed the image script in a way that it would make it much more efficient, it seems it's really fast so far nothing really bad happening but I keep my eyes glued, the change on es traffic is obvious [09:06:58] Amir1: Thanks! Could you link to the CR in case anyone would like to take a look? [09:19:18] sobanski: the patch that enables it is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/698372 but the actual change the script in mwmaint2002 (on a secreen) [09:33:56] 10DBA: db1170 mysql process crashed - https://phabricator.wikimedia.org/T286888 (10jcrespo) [09:34:37] jynus: i saw that ^, i'll handle it in a bit [09:34:48] (it's a bad ram issue) [09:34:57] 👍 [09:35:18] just created the ticket for tracking puposes (alerting-related) [09:36:11] also, +1 for orchestrator. i noticed the failure as soon as i logged into work this morning [09:36:29] (but then had an interview, so didn't get to it yet) [09:36:57] I checked it because it is not immediately known to me if it is a general db or a backup db [09:45:07] Random feature thought, adding a note (task number? Phabricator search with host name? Icinga / Grafana link? All of the above?) in Orchestrator. [09:49:30] sobanski: there is a 'downtime' feature in orchestrator, though it does not appear to be working for me right now. we could in theory put a phab url/task id in the 'reason' field [09:49:54] oh, there we go [09:50:01] the full phab url didn't work, the task id did [09:50:05] I was looking at that one but I assume it'll make the red exclamation mark go away which may not be what we want [09:50:46] sobanski: it stops the node showing up in the global status exclamation thingy, but the node itself is still marked as red [09:51:39] I'll write it up in a low priority task for discussion at some mythical point when we have the time for it ;) [09:57:44] 10DBA, 10Orchestrator: Additional operational metadata in Orchestrator - https://phabricator.wikimedia.org/T286891 (10LSobanski) [09:58:00] 10DBA, 10Orchestrator: Additional operational metadata in Orchestrator - https://phabricator.wikimedia.org/T286891 (10LSobanski) p:05Triage→03Low [10:00:37] 10DBA: db1170 mysql process crashed - https://phabricator.wikimedia.org/T286888 (10LSobanski) p:05Triage→03High a:03Kormat [10:29:14] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10MoritzMuehlenhoff) [11:28:15] 10DBA, 10Toolhub, 10User-bd808: Discuss database needs with the DBA team - https://phabricator.wikimedia.org/T271480 (10LSobanski) Hi @bd808. Here's a set of our standard questions * QPS: * Size: Estimating ~1GB in the first year of operation * DB Name: * User: * Accessed from server (s): k8s * Backup Polic... [13:39:48] 10Data-Persistence-Backup, 10Infrastructure-Foundations, 10SRE, 10bacula, 10netops: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) One interesting effect is that, since the datacenter... [14:26:38] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [14:36:23] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [14:38:44] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [14:40:15] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [16:03:00] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [16:23:59] kormat: for when you have some time please https://gerrit.wikimedia.org/r/c/operations/puppet/+/705045 [16:24:06] 🥺 [16:25:01] Amir1: i'll definitely get to it tomorrow [16:25:09] (or i'm lying to you. find out tomorrow! :) [16:56:44] * Reedy coughs [19:56:14] stay tuned [23:28:34] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Bstorm)