[02:44:22] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on es1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:13:24] federico3: have you done sanitize-wiki? [08:13:45] not yet, why? [08:14:32] Your reply on the email said that you were. I'll do it now then [08:15:38] I can do it in the morning [08:18:04] Why wait? I have started a check run [08:20:17] "argument --section: not allowed with argument --check-only" why is that a restriction? [08:22:05] I'm busy with another thing but I can start it now [08:23:09] Is that in response to the question about why section was added to a mutually exclusive group? [08:23:14] no [08:23:22] for that one I can check in a minute [08:25:04] As I said, I already started a check and then found that I can't check a different section. So, are you doing it or me? [08:25:16] It is private data after all [08:25:50] I'm running it [08:26:30] I'm looking at the code, not sure why --section is in the mutually exclusive group, maybe it could be reviewed at some point, but in general we only run sanitization on s5 [08:27:07] The emails show s3 too. Re the change, you appear to have added it - I would guess that it should have gone on parser and not group [08:27:25] I'm aware [08:28:13] anyhow the sanitization is meant to be idempotent and safe anyways [08:30:01] OK, I am not sure that I questioned that though [08:34:28] if something is showing up on s3, than something is wrong :/ [08:34:59] the wiki is created on s5 [08:35:11] yes, the new wikis are always created in s5 [08:35:44] I'm looking if the script also sees something on s3 but minwikiquote should not be there [08:36:02] example "Private data detected at clouddb1013 for s3 check" [08:36:11] ah, it's a different thing [08:36:29] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1308006 [08:36:48] I looked at the logs, someone needs to run maintain-views on oathauth_user_handles on all.dblist [08:37:30] yep [08:37:40] Manuel is "Fuera de la oficina" right now [08:37:43] I can do it [08:38:10] Amir1: should we discuss in _security instead? [08:41:51] Amir1: thanks! [09:21:28] federico3: are you planning to use test-s4 at all today? [09:21:53] cezmunsta: no, go ahead [09:22:01] +1 [09:29:22] db1208 has a read-only filesystem, known issue? [09:29:52] (noticed it since an update of a globally installed package failed for db1208) [09:31:09] it has a bunch of alerts on alertmanager but it's analytics_meta/matomo [09:31:26] moritzm: you should ask data engineering team [09:33:23] right [09:33:35] these random-one-off DB hosts should really use a different naming scheme [09:35:23] opened https://phabricator.wikimedia.org/T431542 for the DP SREs [09:35:50] what I am not sure is how mysql keeps working with that broken, if it does [09:37:15] on the bright side, it is hard to compromise a RO fs host :-D [09:38:47] also we have a recent backup [10:56:55] Is Puppet supposed to be the only process that makes sure that pt-heartbeat-wikimedia is running? [11:01:16] Also, it would appeat that the Icinga check for that is easily fooled by watching a pgrep -f pt-heartbeat-wikimedia [11:13:02] cezmunsta: no, because it should check there is only 1 process, not 2, alert otherwise [11:13:31] but more importantly, lag will show and alert based on that, wouldn't it? [11:14:14] jynus: PROCS OK: 2 processes with args 'pt-heartbeat-wikimedia' | procs=2;;1:;0; [11:14:23] then that is missconfigured [11:14:51] feel free to send a patch [11:15:03] @cezmunsta some cookbooks stop/start pt-heartbeat as needed but then it runs by itself [11:15:34] It doesn't run by itself unless Puppet starts it from what I see in puppet [11:15:49] .. or of course a call in code, such as the DC switch [11:16:09] jynus: if I stop the heartbeat and run pgrep -f then 1 appears :) [11:17:07] I will check what can avoid that [11:17:47] well, when I designed it, it shouldn't ran from localhost, but from a centralized location to avoid having multiple runs at the same time [11:18:16] but there was no easy way to ensure redundancy at the time [11:18:31] jynus: the check or the heartbeat process? [11:18:39] both [11:19:42] I think a refactor was waiting on having a canonical location for masters [11:21:43] but I insist the important check is the replica lag one, as it checks the result, not the process itself [11:23:38] That should be enough to guarantee pt-heartbeat is working, checking the process is not that useful, as it could be running but failing to write to the table [11:24:30] I am not sure that I quite follow why you would want it centralised? [11:25:04] well, I don't see why we would have 0 processes and the app not exploding [11:25:08] sort of [11:25:13] so not that important [11:25:34] but I see accidentally writing to 2 hosts in parallel by accident due to a bad configuration [11:25:51] and missing the check because "the table is being updated" [11:26:13] so centralizing it (but with redundancy) makes sure it is run just once per section [11:26:49] I guess orchestrator will break in that case [11:27:21] in general there is a lack of a single canonical place where dynamic config is handled, outside of the app itself [11:28:00] so I belive puppet right now fills that gap (badly) [11:28:03] Aside from the sanitarium, doesn't using h=localhost resolve your concern about "wrong host"? [11:28:12] no [11:28:29] Why not? [11:28:39] you could have 2 masters by accident, what prevents you from that? [11:28:53] or 10 [11:29:10] What is the issue with writing a heartbeat to every master? [11:29:31] That tells you how far its children are behind it [11:29:36] define master here. what I have a problem is with writing to a non-master [11:31:09] also it was decided (not by me) that there would be only one active writer at a time, or orchestrator failed [11:31:46] in fact, there is an alert if there is leftover records from a previous master [11:32:23] What is the alert message/check name? [11:32:37] I think orchestrator will say it is lagged [11:32:51] because it can only check based on server_id or something [11:33:17] not sure if also the replication lag too, I think it did when I did the switchover recently [11:33:38] until I cleaned up the old record [11:34:17] so we are agreeing here, there is room for improvement :-D [11:34:38] and it doesn't matter the direction, as long as it is consistent [11:36:06] in my original vision, we had a percona cluster as master for full master redundancy and semi-sync for replicas, so failovers were easier [11:36:17] that may be still on a ticket somewhere [11:37:00] pc has the heartbeat [11:38:12] https://phabricator.wikimedia.org/T172497 [11:38:54] nah, that's for the issues on the app layer [11:38:57] let me search more [11:40:26] but the original idea was to have M-M <---->M-M, where reach M-M was a percona cluster on each datacenters as primary, so if one master failed, the other took over transparently, fixing the SPOF, and each with its own heartbeat [11:41:26] pt-heartbeat could live on kubernetes now tbh [11:42:07] most client, stateless services should live there anyway [11:45:47] aside from redundancy, state should not depend on puppet (static config) [11:46:39] The state is in the database [11:46:53] I mean the dynamic config [11:47:14] what's the master, where to send pt-heartbeat, etc. [11:47:25] so I am agreein with you, we can do better :-D [11:48:24] plus some decisions were ok 20 years ago, but not now, it is ok to question stuff and keep improving 💪 [11:49:11] I found the ticket: https://phabricator.wikimedia.org/T119626 [11:49:43] cezmunsta: I challenge you to setup that :-D, I couldn't [11:57:49] Just use ProxySQL :) [11:58:46] https://www.youtube.com/watch?v=ZXsQAXx_ao0