[05:00:31] No, it is enabled [06:00:32] starting https://phabricator.wikimedia.org/T429868 Switchover s6 master (db1201 -> db1173) [06:53:20] oh ok for some reason I thought it was disabled. then I can do some rigging [06:53:25] *digging [08:54:48] looks like performance_schema is enabled, but not collected by the prometheus exporter, which makes it harder to find what happened [08:55:22] but I did find some heavy activity from user s53003 (tools.xtools-dev) [08:55:54] if you see repeated spikes, ping me again and I'll double check what that tool is doing [08:56:02] so far, it seems to have stopped [09:19:50] dhinus: you could perhaps try pt-stalk along with "--function processlist" to track it for you, maybe using "--variable User" or "--variable Time" etc along with a suitable --match ... unless marostegui knows a reason not to [09:20:57] Something more specific could be done with a file for the function if needed [09:24:50] cezmunsta: Works for me, not sure if we have it installed on clouddb1014, but if not it is of course fine to get it installed [09:24:51] dhinus: ^ [09:30:12] thanks, I'll have a go [09:30:28] it's already installed [09:39:28] never used it before, IIUC it's only useful if we expect other spikes. would you leave it running in a screen session? [09:41:17] honestly I would probably do nothing unless we see other spikes. did the spikes yesterday cause any issues (alerts/pages/etc.)? [09:42:07] dhinus: They made the monitoring system have a glitch, that's how I noticed it [09:42:22] what kind of glitch? :) [09:43:14] 19:09:52 <+icinga-wm> PROBLEM - MariaDB read only wikireplica-s2 on clouddb1014 is CRITICAL: Could not connect to localhost:3312 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [09:43:14] 19:09:52 <+icinga-wm> PROBLEM - MariaDB read only s2 on clouddb1014 is CRITICAL: Could not connect to localhost:3312 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [09:43:16] dhinus: ^ [09:43:49] ack! [09:43:51] dhinus: that xtools sounds familiar [09:43:57] didn't that cause issues before? [09:44:24] I don't remember recent issues, but it does sound familiar [09:44:51] Found it, it triggered: https://jira.mariadb.org/browse/MDEV-39209 [09:45:47] ah the big queries that caused the crashes, yes now I remember! [09:45:52] yeah [09:46:10] every time I see: https://jira.mariadb.org/secure/attachment/77131/query.txt i cry.. [09:46:31] :/ [09:49:23] I can try to engage with the developers, but as long as we let users run SQL they will come up with weird things... [09:49:38] yeah, that wasn't their fault to be fair [09:50:03] Mariadb shouldn't allow those if they cannot cope with them [09:50:08] the server shouldn't crash that way [09:50:45] toolforge users are basically testing the edge cases of mariadb :P [09:50:54] dhinus: you can use --daemonize and --log and leave it running, plus if you use --mysql-only then it avoids the extra info that you probably aren't interested in for this scenario [09:51:01] it was very easy to reproduce, so it was a good one [09:56:42] cezmunsta: thanks. would you use --variable=time or --function=cpu as a filter? [09:58:24] what was it that you checked when you found "heavy activity"? [10:10:08] I checked performance_schema.events_statements_summary_by_user_by_event_name, and found that user has high values for SUM_TIMER_WAIT and SUM_SELECT_SCAN, but I'm not at all sure if that's a good indicator... [10:10:21] the real thing we want to find is the CPU spikes we saw yesterday [10:12:56] gemini was telling me I could pass --function=cpu to pt-stalk but I don't see it in the docs, so that was probably hallucinated :( [10:14:46] I would say that is probably true ... set_trg_func permits a file or processlist | status ;) [10:14:57] Just testing a script for you [10:15:41] thanks, anyways I think we should not spend too much time on this, maybe it was a one-off thing... [10:15:58] if you can craft a pt-stalk that makes sense, we can run it for a few days [10:16:46] otherwise we can also ignore this until it happens again :) [10:17:01] I will also check if we can route the icinga alerts to my team, or to both teams [10:26:12] opened T429892 to track this [10:26:12] T429892: CPU spikes in clouddb1014@s2 - https://phabricator.wikimedia.org/T429892 [10:45:09] should we consider exporting performance_schema to prometheus, at least for clouddbs? maybe just --collect.perf_schema.users? [10:47:38] hmm that's not the right collect flag [10:53:12] Maybe --collect.sys.user_summary ? [11:01:04] yes that might work! [11:08:58] the sys.user_summary view exists and is also very handy to use in SQL (compared to querying performance_schema directly) [11:10:12] for some reason it's not listed in https://mariadb.com/docs/server/reference/system-tables/sys-schema/sys-schema-views [11:11:48] dhinus: that's probably old [11:11:54] I think user_summary has been there for years though [11:12:16] from 10.6 apparently [11:12:43] WDYT about collecting it in prometheus? [11:14:37] * dhinus lunch, bbl [11:20:07] dhinus: I would say that it might be worth it, you can always disable it again later if need be. I guess that you could even then use the exporter as a source for a pt-stalk function script by hitting the metrics endpoint [11:24:05] federico3: for the rolling restarts for the kernel, has s4 been done yet or still in progress? [11:26:57] There are a few nodes that have a long enough uptime to make me ask :) [11:34:40] FIRING: SystemdUnitFailed: pt-heartbeat-wikimedia.service on db1290:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:36:47] ^ expected [11:36:49] I am setting it up [11:57:10] cezmunsta: I'll check in a bit [11:57:46] ack thanks [12:02:29] but in general the security task is up to date [13:45:03] cezmunsta: dhinus: security prohibited enabling those for production specifically into public prometheus for production for leaking user activity. I cannot speak for wmcs. [13:45:56] jynus: ah interesting point. I need to check exactly what information is published, but potentially we might have the same issue [13:46:48] I remember specifically it worried the team a the time for being able to correlate encrypted TLS activity and wiki edits, with such a granularity [13:47:28] I don't have the context for mysql @ wmcs, it may not be an issue there [13:48:21] if prometheus ends up not public, or we setup a private instance for private data, that could go there [13:48:58] there is a ticket for that dhinus that IMHO should be unbreak now: https://phabricator.wikimedia.org/T388813 [13:55:47] dhinus: just to be clear, that extra info we *really really* need it, and was initially enabled but turned off soon after, and we need a way to reenable it privatelly back again [13:56:32] or setup another way to scrape it and aggregate it [13:57:15] jynus: thanks for the context! :) I don't think we //really// need it for WMCS, it would be handy in a few situations but we can live without it [13:57:32] obs also has been worried about the scaling of certain features, so we need to work with them [13:57:37] if you end up enabling it in prod (with private prometheus), we should definitely enable it in WMCS as well [13:58:10] but I'm not sure I want to have it for WMCS only, at least I will check with the security team first [13:58:20] because metrics on wmcs goes to the same prometheus stack, right? [13:58:39] in theory we have less sensitive data, but we might still have some. yes same stack for clouddbs. (for toolforge, different stack) [13:59:16] there is a simple solution.. albeit with a downside... it is a view and you can redefine that view with a hash around the user field [14:05:34] cezmunsta: that might work... I also re-checked the sys.user_summary view and it does not contain any sensitive info apart from the user name [14:05:56] the full performance_schema might be more tricky if it contains actual queries/processlist (does it? I'm not sure) [14:07:14] I don't think db users are very worying on production, as all app queries run through a couple of the same users [14:08:14] I also don't have a lot of context on the expectation of privacy of wmcs db users, my suggestion is to create a ticket and ask for help auditing, be transparent to users about changes [14:09:20] I also don't know the cardinality of text-based metric tags (are there a million users?), obs usually complains about that [14:10:07] so my suggestion is: create a task and add a few key people there so everything happens transparently [14:12:13] dhinus: re performance_schema.. I think that it only queries performance_schema.events_statements_summary_by_digest and so you get placeholders ... BUT in theory I guess that you could see undesired info in there, e.g. a DEFINER when creating a view etc [14:13:22] it can be enabled on a host as a test but not scrape it, then audit it thoroughly [14:16:06] I'll open a separate task for clouddb, so it can be discussed separately, and link it to the prod one [15:55:00] Amir1: are you creating wikis today? [15:55:14] yeah if that's fine [15:55:24] It can wait, no rush on my side [15:55:24] Amir1: yep, no problem, so we'll expect the alerts then tomorrow [15:55:39] Amir1: no no, no problem at all, I just saw you in -operations and I was wondering [19:17:22] the task about getting sys.user_summary in prometheus: T429993 [19:17:22] T429993: [wikireplicas] Publish sys.user_summary to Prometheus - https://phabricator.wikimedia.org/T429993