[00:10:36] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:10:36] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:16:33] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) [07:06:07] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) [07:30:10] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) [07:44:22] (SystemdUnitFailed) firing: (2) bird.service Failed on ganeti2033:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:52:33] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10Marostegui) db2105 is no longer a master. This host can be done after being depooled [07:52:43] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10Marostegui) [07:53:31] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) >>! In T355862#9494195, @Marostegui wrote: > db2142 - x2 master > db2103 - s1 master > es2020 - es4 master The... [08:23:40] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ayounsi) To keep on the radar but non-blocking : {T309724} [08:41:10] slyngs: FYI puppet is failing on sso-debmon.sso.eqiad1.wikimedia.cloud, I guess is due because of the last refactor, maybe something missing in hiera? [08:41:13] Could not find resource 'Service[uwsgi-debmonitor]' in parameter 'notify' (file: /etc/puppet/modules/profile/manifests/debmonitor/server.pp, line: 100) [08:41:27] the service is actually there and running but maybe puppet is not anymore aware of it? [08:42:13] volans: Thanks, I've missed that when doing the refactoring [09:32:52] volans: fixed [09:33:02] slyngs: thanks! [09:33:24] I missed that Debmonitor is installed via... not scap3 [09:33:37] yeah [09:33:50] Manually installed ? [09:35:44] no idea [09:36:04] I think that was create by joh.n or mor.itz [09:36:08] not me :-) [09:36:30] Sure, blame it on the guy that left, classic :-) [09:36:36] ofc :D [09:36:37] but when the deb-based deployment is completed in prod, we can also re-install with the deb as well [09:36:52] and remove support for non-deb deployments [09:36:54] in puppet [09:36:59] definitely! [09:37:43] add add a CI hook which makes the patch -2 if only the word scap is added to modules/debmonitor [09:38:01] :D [09:38:14] I +1 your -2 :-P [09:39:11] or send a PR to Gerrit to allow -3 specifically for this use case :-) [09:43:37] While on the topic of debmonitor, everything seems okay, but is there a way to force my own session onto the new debmonitor server? [09:44:08] -s SERVER, --server SERVER [09:44:39] I was thinking the UI :-) [09:44:57] then I didn't get the question, sorry [09:45:00] what do you mean? [09:45:43] he means that debmonitor.wikimedia.org gets served by 1002 and if there is a way to make it use 1003 [09:45:59] we can either add a debmonitor-staging.w.o [09:46:00] ssh tunnel and browser to localhost? [09:46:07] debmonitor-next.w.o [09:46:10] etc.. [09:46:14] and after the update also make it point to 1003 [09:46:18] or yes, SSH tunnel [09:46:49] there hasn't been a huge amount of work on debmonitor after the initial setup [09:47:11] but given that there will be the need for more feature work for better handling of container images [09:47:30] maybe eventually having a staging host as well does make sense [09:47:36] if we plan to have a real -next instance like netbox [09:47:49] we need to setup one, let's not point -next and prod to the same host :D [09:47:56] and as a first step we could just as well start with a -next for now and initially have it point to the 2003s [09:47:59] yeah [09:48:16] what about the DB? [09:48:22] is it using a different db or the same of prod? [09:48:29] idp-test and idm-test have also been immensely helpful over the years [09:48:52] for IDP we have a different DB [09:49:18] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10klausman) [09:49:25] 10Mail, 10Infrastructure-Foundations: Received 4 mail notifications but only 2 actual mails - https://phabricator.wikimedia.org/T351027 (10AlexisJazz) 05Open→03Declined @jhathaway nope. Guess it was a fluke, or one of the Wikimedia mail servers (assuming that's load balanced) was malfunctioning, or maybe C... [09:49:28] for IDM-test most things are in LDAP anyway, but the local DB is different IIRC [09:49:57] so for a fullblown debmonitor-test we would use a new separate DB I guess [09:50:17] for 1003/2003 we can simply have both use the same DB AFAICT [09:50:34] the data is populated by the ingestion endpoint anyway [09:50:35] yeah I'd rather use a separate db [09:51:06] separate DB also makes it simpler to experiment with layout changes for container images [09:51:10] That would also allow us to test database migrations [09:51:16] like if we start to track more meta data [09:51:42] but for the new bookworm hosts [09:51:53] given that the primary target audience is SREs [09:52:15] we can also be bold and just failover to the new hosts, check them and failing that, revert and fix up [09:52:43] and add debmonitor-next when there's actually a proper staging host [09:52:51] wfm [09:57:44] 10netbox, 10Infrastructure-Foundations: Evaluate usage of Kubernetes/Wikikube Tags in netbox and replace them with something if possible - https://phabricator.wikimedia.org/T354169 (10akosiaris) Thanks. I am not rush on my side as well, so +1 on waiting for the upgrade. [10:17:18] XioNoX, topranks: puppet is disabled since long time on netbox-dev2002 with reason: test-swift-netbox - root [10:17:29] was it left disabled? can it be re-enabled? [10:17:36] that was me [10:17:40] yeah you can re-enable it [10:17:46] Weeeee: INFO:debmonitor:Successfully sent the full update to the DebMonitor server [10:18:03] XioNoX: ack thx {done} [10:23:45] and now to the status of /srv/deployment/netbox-extras on netbox-dev [10:23:50] what should I do with local changes? [10:29:20] volans: save a git diff somewhere on the side and reset it [10:30:06] I can also try to stash and re-apply, but I wonder if that's needed [10:34:44] mmmh cumin1002 can't ssh to netbox-dev2002.codfw.wmnet, acls? [10:34:53] telnet gets stuck [10:34:56] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [10:38:08] moritzm: does it ring any bell? I can see the rule in nf list ruleset: [10:38:11] ip6 saddr @CUMIN_MASTERS_ipv6 tcp dport { 22 } accept [10:38:12] and same for ipv4 [10:38:21] and the variable has the IP of cumin1002 [10:59:22] (SystemdUnitFailed) firing: (3) apache2.service Failed on debmonitor1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:13:30] 10netops, 10Data-Persistence, 10Data-Persistence-Backup, 10Infrastructure-Foundations, and 3 others: Migrate servers in codfw rack B4 from asw-b4-codfw to lsw1-b4-codfw - https://phabricator.wikimedia.org/T355860 (10MatthewVernon) [11:14:14] 10netops, 10Data-Persistence, 10Data-Persistence-Backup, 10Infrastructure-Foundations, and 3 others: Migrate servers in codfw rack B4 from asw-b4-codfw to lsw1-b4-codfw - https://phabricator.wikimedia.org/T355860 (10MatthewVernon) [I'll want to check afterwards that the ms-be nodes are happy, but this shou... [11:16:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A2 from asw-a2-codfw to lsw1-a2-codfw - https://phabricator.wikimedia.org/T355861 (10MatthewVernon) [11:17:55] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [11:19:06] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A2 from asw-a2-codfw to lsw1-a2-codfw - https://phabricator.wikimedia.org/T355861 (10MatthewVernon) swift will need depooling in codfw before this work. Likewise the affected thanos-fe node. I... [11:19:11] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10MatthewVernon) [11:21:20] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10MatthewVernon) Once complete, I'll want to check the ms-be nodes are all happy (shouldn't be an issue). [11:21:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A7 from asw-a7-codfw to lsw1-a7-codfw - https://phabricator.wikimedia.org/T355867 (10MatthewVernon) [11:22:18] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A7 from asw-a7-codfw to lsw1-a7-codfw - https://phabricator.wikimedia.org/T355867 (10MatthewVernon) Once complete I'll want to check the backends, but this shouldn't be an issue. [11:23:22] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B2 from asw-b2-codfw to lsw1-b2-codfw - https://phabricator.wikimedia.org/T355868 (10MatthewVernon) [11:25:04] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B2 from asw-b2-codfw to lsw1-b2-codfw - https://phabricator.wikimedia.org/T355868 (10MatthewVernon) The affected thanos frontend will need depooling. Similarly, swift in codfw will need depooling. [11:26:33] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872 (10MatthewVernon) [11:27:35] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872 (10MatthewVernon) I'll want to check the backends once this work is complete, but it shouldn't be an issue. [11:36:20] volans: having a look [11:36:50] thx [11:45:16] strange, it doesn't seem to be related to nftables; if I e.g. do systemctl stop nftables (and doublechecked with "nft list ruleset" to be empty), the connection still fails [11:45:20] I'll poke more [11:46:30] doh [12:55:28] trace id c600d76b inet base input packet: iif "ens13" ether saddr 64:87:88:f2:73:c9 ether daddr aa:00:00:d8:a8:d5 ip6 saddr 2620:0:861:107:10:64:48:98 ip6 daddr 2620:0:860:104:10:192:48:191 ip6 dscp cs0 ip6 ecn not-ect ip6 hoplimit 61 ip6 flowlabel 199452 ip6 nexthdr tcp ip6 length 40 tcp sport 55750 tcp dport 22 tcp flags == syn tcp window 43200 [12:55:30] trace id c600d76b inet base input rule ip6 saddr @CUMIN_MASTERS_ipv6 tcp dport { 22 } meta nftrace set 1 accept (verdict accept) [12:55:31] trace id a9507e85 inet base input packet: iif "ens13" ether saddr 64:87:88:f2:73:c9 ether daddr aa:00:00:d8:a8:d5 ip6 saddr 2620:0:861:107:10:64:48:98 ip6 daddr 2620:0:860:104:10:192:48:191 ip6 dscp cs0 ip6 ecn not-ect ip6 hoplimit 61 ip6 flowlabel 624959 ip6 nexthdr tcp ip6 length 40 tcp sport 55750 tcp dport 22 tcp flags == syn tcp window 43200 [12:55:33] trace id a9507e85 inet base input rule ip6 saddr @CUMIN_MASTERS_ipv6 tcp dport { 22 } meta nftrace set 1 accept (verdict accept) [12:55:34] trace id 50d4f8bd inet base input packet: iif "ens13" ether saddr 64:87:88:f2:73:c9 ether daddr aa:00:00:d8:a8:d5 ip6 saddr 2620:0:861:107:10:64:48:98 ip6 daddr 2620:0:860:104:10:192:48:191 ip6 dscp cs0 ip6 ecn not-ect ip6 hoplimit 61 ip6 flowlabel 422042 ip6 nexthdr tcp ip6 length 40 tcp sport 55750 tcp dport 22 tcp flags == syn tcp window 43200 [12:55:36] trace id 50d4f8bd inet base input rule ip6 saddr @CUMIN_MASTERS_ipv6 tcp dport { 22 } meta nftrace set 1 accept (verdict accept) [12:55:37] trace id 26354226 inet base input packet: iif "ens13" ether saddr 64:87:88:f2:73:c9 ether daddr aa:00:00:d8:a8:d5 ip6 saddr 2620:0:861:107:10:64:48:98 ip6 daddr 2620:0:860:104:10:192:48:191 ip6 dscp cs0 ip6 ecn not-ect ip6 hoplimit 61 ip6 flowlabel 652244 ip6 nexthdr tcp ip6 length 40 tcp sport 55750 tcp dport 22 tcp flags == syn tcp window 43200 [12:55:39] trace id 26354226 inet base input rule ip6 saddr @CUMIN_MASTERS_ipv6 tcp dport { 22 } meta nftrace set 1 accept (verdict accept) [12:55:40] trace id ccba55e0 inet base input packet: iif "ens13" ether saddr 64:87:88:f2:73:c9 ether daddr aa:00:00:d8:a8:d5 ip saddr 10.64.48.98 ip daddr 10.192.48.191 ip dscp cs0 ip ecn not-ect ip ttl [12:55:42] 61 ip id 61714 ip protocol tcp ip length 60 tcp sport 55288 tcp dport 22 tcp flags == syn tcp window 42340 [12:55:43] trace id ccba55e0 inet base input rule ip saddr @CUMIN_MASTERS_ipv4 tcp dport { 22 } meta nftrace set 1 accept (verdict accept) [12:55:45] trace id 18cddaad inet base input packet: iif "ens13" ether saddr 64:87:88:f2:73:c9 ether daddr aa:00:00:d8:a8:d5 ip saddr 10.64.48.98 ip daddr 10.192.48.191 ip dscp cs0 ip ecn not-ect ip ttl [12:55:46] 61 ip id 61715 ip protocol tcp ip length 60 tcp sport 55288 tcp dport 22 tcp flags == syn tcp window 42340 [12:55:48] trace id 18cddaad inet base input rule ip saddr @CUMIN_MASTERS_ipv4 tcp dport { 22 } meta nftrace set 1 accept (verdict accept) [12:55:49] trace id 4fa8b2bd inet base input packet: iif "ens13" ether saddr 64:87:88:f2:73:c9 ether daddr aa:00:00:d8:a8:d5 ip saddr 10.64.48.98 ip daddr 10.192.48.191 ip dscp cs0 ip ecn not-ect ip ttl [12:55:51] 61 ip id 61716 ip protocol tcp ip length 60 tcp sport 55288 tcp dport 22 tcp flags == syn tcp window 42340 [12:55:52] trace id 4fa8b2bd inet base input rule ip saddr @CUMIN_MASTERS_ipv4 tcp dport { 22 } meta nftrace set 1 accept (verdict accept) [12:55:54] trace id 2ef7a032 inet base input packet: iif "ens13" ether saddr 64:87:88:f2:73:c9 ether daddr aa:00:00:d8:a8:d5 ip saddr 10.64.48.98 ip daddr 10.192.48.191 ip dscp cs0 ip ecn not-ect ip ttl [12:55:55] 61 ip id 61717 ip protocol tcp ip length 60 tcp sport 55288 tcp dport 22 tcp flags == syn tcp window 42340 [12:55:57] trace id 2ef7a032 inet base input rule ip saddr @CUMIN_MASTERS_ipv4 tcp dport { 22 } meta nftrace set 1 accept (verdict accept) [12:55:58] sorry, mispaste [12:56:00] It's quite puzzling, I'm looking further, initial findings collected at https://phabricator.wikimedia.org/T356174 [13:05:17] thx, I got lost in the mispaste :D [13:08:59] the working connectiona dn the failing ones have identical traces, just the failing one is repeated multiple times over v6 and v4 [13:10:08] fyi, I fixed one of the problematic server by rebooting it last week iirc [13:10:16] forgot which one though [13:14:01] for the failing one, the additional attempts are simply caused by cumin's retries [13:16:53] moritzm: could there be some iptables leftovers blocking the flow ? [13:17:02] like iptables still have the old rule set [13:20:21] the old ruleset was allowing it though... [13:20:37] but yes if the reboot resolve it could possibly be something like that [13:20:55] yeah, and e.g. ganeti-test* was one of the first roles to have switched to nftables [13:22:08] can we install iptbles binary and see if there is anything or will that be in conflict with nftables [13:22:20] volans: my theory was that the "old" ruleset was the one before cumin1002 was created [13:22:26] and e.g. netmon was only recently switched to nfables and has no issues [13:22:36] I don't think we can see the rules in kernel's memory directly [13:22:42] XioNoX: got it, then makes sense [13:23:06] but if switching to nftables means that old rules still are in place until rebooted we're doing something wrong in the migration : [13:23:08] I'll doublecheck just in case (wrt iptables -L) [13:23:09] :D [13:23:33] it's entirely possible that nftables is just a red herring [13:24:27] and iptables on netbox-dev2001 is in fact empty as well [13:24:36] and "iptables -L" on netbox-dev2001 is in fact empty as well [13:26:00] I'll poke at it some more [13:26:54] we could check the uptime and when cumin1002 was added to nftables rule [13:27:11] and see if all of them have an uptime higher than that [13:27:15] *sorry to iptables [13:29:11] there's no correlation of that sort: idm-test1001 has an uptime far longer than cumin1002 exists and works fine e.g. [13:30:34] I'll try if I can setup some traces speifically on the outbound connections, maybe it gets accepted, but the return of the SSH handshake to 1002 fails [13:31:45] ack thx [13:32:39] but I'd like to keep netbox-dev2002 in that state for debugging, let's just use cumin2002/1001 for now to access it [13:33:00] sure, no prob [13:45:26] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by ayounsi@cumin1002 for hosts: `srestest2005.codfw.wmnet` - srestest2005.codfw.wmnet (**WARN**) - //Host... [14:12:48] XioNoX, topranks: I've deployed the last merged change in netbox-extras to netbox-dev, and then re-applied the local changes that were present. I'll leave it to you to clean them up once done with testing or when they are merged. [14:16:41] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: WMCS VIPs: Netbox netmask inconsistencies - https://phabricator.wikimedia.org/T295774 (10Volans) p:05Triage→03Medium @cmooney did you had a chance to test the above failure scenario? AFAICT is still happening [14:21:30] thx! [14:56:27] 10netbox, 10Data-Engineering, 10Data-Persistence-Backup, 10Infrastructure-Foundations, 10bacula: Convert Netbox data (PostgresQL) longterm storage backups (bacula) into full backups rather than incrementals - https://phabricator.wikimedia.org/T316655 (10Volans) Adding #data-engineering as the change will... [15:00:36] (SystemdUnitFailed) firing: (2) envoyproxy.service Failed on debmonitor1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:07:31] I have a few more leads (still to be confirmed), but it seems like act we'll need to enforce a reboot after the transition towards the nftables provider [15:08:49] notably the fw-in-drop ulogd classifier only ever get/got configured for ferm, but still logs dropped traffic on the affected hosts [15:09:22] moritzm: do we clear ferm/iptables before migrating? resetting all rules I mean [15:10:18] ferm gets stopped, which flushes the he rules [15:10:33] are we sure it does it? :D [15:10:35] but e.g. I have been unable to far to rmmod nfnetlink_log/nfnetlink [15:10:40] sorry for asking obviously stupid questions [15:11:15] yeah, the mere stopping has always worked reliably, that shouldn't be an issue [15:12:11] possibly some functionality gets influenced by the presence of loaded kmods and always assuming a reboot is the only reliable way to reset this [15:12:24] I'll dig some more tomorrow with a summary on task [15:17:08] thanks a lot, sorry for the additional unplanned work on this [19:00:36] (SystemdUnitFailed) firing: (2) envoyproxy.service Failed on debmonitor1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:13:55] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Epic: Move most (all?) exim personal aliases to WMF ITS - https://phabricator.wikimedia.org/T122144 (10Dzahn) [20:03:10] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [23:00:36] (SystemdUnitFailed) firing: (2) envoyproxy.service Failed on debmonitor1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed