[05:13:12] I am going to start removing eqiad -> codfw replication [05:45:15] That is done, maintenance can resume! [08:01:46] hey, just a head's up before monday: lenovo tech support is supposed to come to swap my laptop's motherboard, I have no idea when they should be doing so. I might be attending our weekly on my phone, so sorry in advance for the general sound quality! [08:03:14] that laptop didnt last long [08:04:00] bad luck I guess, a firmware upgrade went wrong and after that it was all downhill haha [08:04:57] And wait until you change your kernel! [08:06:57] :( [08:08:48] btw https://phabricator.wikimedia.org/P52549 [08:08:48] Average time of each request spending time waiting for the db, split by day and end point [08:10:38] we can make it faster by truncating some tables [08:11:02] xD [09:25:05] are there obvious patterns there I should be noticing? [10:13:20] it is quite stable, I could use it as SLO [10:20:36] 😨 [10:52:11] schema migrations will start again on s7 [15:00:45] jynus, re: performance_schema it's enabled (p_s = 'on' interpolation to performance_schema = 1 in .cnf). andrewbogott i think it'd need to be done like https://mariadb.com/kb/en/performance-schema-overview/ and https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/prometheus/manifests/mysqld_exporter.pp or variant tweaked in order for prometheus-mysqld-exporter to get it over to [15:00:45] https://grafana.wikimedia.org/d/000000273/mysql?from=now-7d&to=now&var-job=mysql-labs&var-server=clouddb1019&var-port=13314&orgId=1&refresh=1m (e.g., in "Potential contention causes" section, I think). [15:00:50] or just allow plain querying. but, i think there may need to be consideration for, ironically, performance impacts if enabling this for a window or in perpetuity. according to dpkg it looks like we're at version 0.12.1 of prometheus-mysqld-exporter, so you could look at some of the queries in perf_*.go in https://github.com/prometheus/mysqld_exporter/tree/v0.12.1/collector for inspiration should this sort of thing become [15:00:50] necessary. [15:00:52] not sure marostegui Amir1 what you think about the notion of having this turned on by default, maybe for clouddb1013-1020, anyway? (i can't imagine we'd want it on 1021, as with any of our mediawiki app server-connected DBs). maybe better to leave that to another day. just writing stuff down here to get some info down and curious to hear your thinking. [15:01:44] dr0ptp4kt: security explicitly told us not to make that public [15:03:04] but we can talk on Monday, when I come back [15:05:52] yeah, we possibly can even fully turn off p_s to gain some memory in clouddbs [15:15:54] marostegui arnaudb: FYI, I'm running the pagelinks schema change on master of eqiad with replication [15:16:00] in s8 and s1 [15:16:13] noted [15:16:18] double checked and eqiad is fully depooled [16:05:38] ah, thanks for the context jynus. thanks also Amir1. i realize having it (and with the full config activated - not sure if that's the case, i don't have root) versus completely turning it off are at odds, but it does seem like a good topic! [16:06:05] jynus have a good break! [16:07:01] (sorry, didn't mean to cause irc ping j.ynus, didn't think to look at the calendar) [18:35:46] https://github.com/wikimedia/mediawiki-extensions-WikimediaMaintenance/blob/master/storage/make-all-blobs 😭 [18:45:20] what's up? [18:47:04] it's a bash that reads password from calling php and then runs mysql with that password opening a sql file and replacing the table name [18:47:19] surely there must be a better way to do this [18:48:37] well yeah, but I'm sure it was done like that for a reason [18:54:56] I'll see what I can do about it