[07:35:06] Deployed, checked some backends, looks OK... [08:29:07] Backup auto-healing (sort of) working: https://phabricator.wikimedia.org/T293975#7817389 [08:40:51] \o/ [11:39:33] marostegui: 50GB removed for s3 for flaggedtemplates alter table https://grafana.wikimedia.org/d/000000377/host-overview?viewPanel=28&orgId=1&var-server=db1157&var-datasource=thanos&var-cluster=wmcs&from=now-1d&to=now [11:39:52] (it can be also triggering optimize and clean ups kicking in but still!) [11:39:59] yeah, pretty nice [11:47:41] Amir1: not manuel, but I am guessing a --help when run without parameters (a proper argparse) [11:48:15] aah, yeah makes sense [11:48:33] jynus: hehe yeah [11:48:36] just replied [12:00:28] Emperor: well done re: swift ring management deployment ! [12:05:26] marostegui: slightly OT, we really should clean up content of dbtools [12:05:40] yep, I know [14:01:35] marostegui: orchestrator db moved from db2093 to db1115: T301315 [14:01:35] T301315: Move orchestrator from db2093 to db1115 - https://phabricator.wikimedia.org/T301315 [14:01:44] and it looks like i didn't screw it up! 😅 [14:05:28] \o/ [14:06:03] good! [14:06:55] there's a whole bunch of stuff on this section that could be nuked [14:07:04] e.g. a `percona` db on db1115 that was last used in 2015 [14:12:04] oh, oof. db2093 is on `partman/manual-setup.cfg` 🥀 [14:14:26] yep, I had fun doing the reimage of db1115 [14:16:21] kormat: this is what I did https://gerrit.wikimedia.org/r/c/operations/puppet/+/761527 [14:21:04] i did it manually, like a peasant. [14:24:47] https://wikitech.wikimedia.org/wiki/Swift/Ring_Management first pass at a wikitech page [14:56:46] marostegui: this doesn't look friendly: https://phabricator.wikimedia.org/P23832 [15:01:51] that's an orchestrator query [15:02:40] seems like the answer is to switch the binlog format to ROW [15:07:33] as no adults stopped me, i made the change live [15:12:14] aw, the incident ritual feedback form asks for middle-endian dates [15:12:50] eh? [15:13:47] ah, the google form [15:14:03] yes [15:14:51] it's dd/mm/yyyy for me [15:21:54] Huh. [15:27:50] I have changed my language from English to English (UK) maybe that will help next time :) [15:32:09] marostegui: Did you happen to scare up a dba to work with me on T301949 during your meeting this week? [15:32:09] T301949: ToolsDB upgrade => Bullseye, MariaDB 10.4 - https://phabricator.wikimedia.org/T301949 [15:32:30] andrewbogott: Not yet, I'll get back to you on that [15:32:44] ok! thx :) [16:55:31] PROBLEM - MariaDB sustained replica lag on m2 on db2078 is CRITICAL: 94.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2078&var-port=13322 [16:57:37] RECOVERY - MariaDB sustained replica lag on m2 on db2078 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2078&var-port=13322 [17:30:54] PROBLEM - MariaDB sustained replica lag on m2 on db1117 is CRITICAL: 86.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13322 [17:32:10] RECOVERY - MariaDB sustained replica lag on m2 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13322 [18:56:26] jynus: marostegui kormat I can't find the ticket for central place for grants. I built this today https://people.wikimedia.org/~ladsgroup/omg/ [18:56:49] (wmf ldap group required) [20:43:23] PROBLEM - MariaDB sustained replica lag on m2 on db1117 is CRITICAL: 1051 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13322 [20:47:33] PROBLEM - MariaDB sustained replica lag on m2 on db2078 is CRITICAL: 474.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2078&var-port=13322 [20:47:51] RECOVERY - MariaDB sustained replica lag on m2 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13322 [20:51:59] RECOVERY - MariaDB sustained replica lag on m2 on db2078 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2078&var-port=13322