[07:34:42] marostegui: Can I go ahead and create the db in m5 for T306963? [07:34:43] T306963: Integrate new section mapping database - https://phabricator.wikimedia.org/T306963 [07:43:31] yeah [08:00:28] does that need backups? [08:01:08] jynus: good q. I don't think so because it doesn't get updated and I'm just inserting data from a dump they provided [08:02:03] "nice database, shame if anything were to happen to it" ;-) [08:07:17] :D [08:23:27] Amir1: feel free to offer it for easy recovery, even if it can be generated again, 472MB is not a big size for backups [09:17:23] I was checking motd for a misc server and I got: [09:17:38] db2151 is a Misc DB Server (mariadb::core) [09:18:08] issue seems to be 76a1bfb331a (modules/role/manifests/mariadb/misc.pp) could you check if that is intended? [09:18:30] I can send a patch if not [09:20:22] I think I will just send a patch and add you as reviewers, will work better even to just get all context [09:28:33] sent https://gerrit.wikimedia.org/r/c/operations/puppet/+/798467 but it is not urgent at all, but on first look seems like a silly mistake [10:49:34] Amir1: sorry, I cannot find the database maintenance page. Can I restart db1150 ? [10:49:58] I saw it had an existing downtime [10:50:58] I saw no current ongoing maintenance, but wanted to check with you in case it automatically start soon [10:51:47] jynus: https://wikitech.wikimedia.org/wiki/Map_of_database_maintenance [10:51:51] let me check [10:51:59] thanks, I couldn't find it with search :-( [10:52:17] FYI reboot will only take 15 minutes [10:52:40] but it is not "urgent", I can wait [10:53:43] jynus: hmm, I can't tell exactly if it's needed or not, if it's downtimed but the schema change is done then it should be safe (my other worry is to avoid depooling too many replicas but this is not pooled in the first place) [10:54:08] yeah, it doesn't serve mw requests [10:54:16] let me see if it has been already done [10:54:46] Thanks! [10:57:59] yeah, it has been done already on s4 [10:58:20] cool [10:58:22] also the bot reverted my categorization :-D https://wikitech.wikimedia.org/w/index.php?title=Map_of_database_maintenance&type=revision&diff=1983294&oldid=1983293 [10:58:39] So will quickly restart this backup source [10:58:44] haha, yeah [10:59:04] I will work on making it more accessible [10:59:14] no worries [10:59:22] it is done on s5 too [10:59:34] will quickly restat the server and it will be back up very soon [11:01:01] in any case, I always look if there is ongoing process before a restart [11:16:01] done with db1150 [11:16:29] it is back up [12:08:34] marostegui: I asked about the cxserver grants in IRC, that's how I got it btw [12:09:16] let's get them pasted on the task, for future references [15:45:19] adding data in m5 [16:05:12] PROBLEM - MariaDB sustained replica lag on m5 on db1117 is CRITICAL: 118 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13325 [16:05:38] PROBLEM - MariaDB sustained replica lag on m5 on db2078 is CRITICAL: 113.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2078&var-port=13325 [16:11:58] RECOVERY - MariaDB sustained replica lag on m5 on db1117 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13325 [16:12:24] RECOVERY - MariaDB sustained replica lag on m5 on db2078 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2078&var-port=13325 [16:14:58] this is me [16:15:05] 10k rows per second [16:15:12] that's not that much