[01:34:17] (SystemdUnitFailed) firing: (4) krb5-admin-server.service Failed on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:32:48] (SystemdUnitFailed) firing: (6) httpbb_kubernetes_mw-api-ext_hourly.service Failed on cumin1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:32:48] (SystemdUnitFailed) firing: (6) httpbb_kubernetes_mw-api-ext_hourly.service Failed on cumin1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:47:48] (SystemdUnitFailed) firing: (4) krb5-admin-server.service Failed on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:52:48] (SystemdUnitFailed) firing: (4) krb5-admin-server.service Failed on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:03:09] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10Marostegui) [07:07:33] https://netboxlabs.com/blog/announcing-the-netbox-labs-series-a looks like they're doing well [07:08:48] jbond, moritzm, any idea what's up with "krb5-admin-server.service Failed on krb2002" from a quick look on that channel that's #1 spam :) [07:09:39] then httpbb_kubernetes_mw which not related to our team I guess but here because it's a check on the cumin hosts? godog is there a way to have it alert on the service ops chan instead? [07:09:48] (or at least not here) [07:10:31] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10Marostegui) [07:32:48] (SystemdUnitFailed) firing: (7) httpbb_hourly_appserver.service Failed on cumin1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:34:17] (SystemdUnitFailed) firing: (10) httpbb_hourly_appserver.service Failed on cumin1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:37:53] (SystemdUnitFailed) firing: (12) httpbb_hourly_appserver.service Failed on cumin1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:57:48] (SystemdUnitFailed) firing: (12) httpbb_hourly_appserver.service Failed on cumin1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:10:41] XioNoX: krb2002 is expired downtime, It'll resolve when I merge the patch to make it a full KDCin our realm, maybe tomorrow, not sure. will downtime again [08:32:48] (SystemdUnitFailed) firing: (9) httpbb_hourly_appserver.service Failed on cumin1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:34:17] (SystemdUnitFailed) firing: (9) httpbb_hourly_appserver.service Failed on cumin1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:38:02] (SystemdUnitFailed) firing: (9) httpbb_hourly_appserver.service Failed on cumin1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:42:36] 10netops, 10Infrastructure-Foundations, 10SRE: Juniper RA receive bug CVE-2023-28981 - https://phabricator.wikimedia.org/T334916 (10cmooney) p:05Triage→03Low [09:20:24] XioNoX: i belive krb2002 is currently been built and prepared to be made live [09:20:49] ahh see mor.itzm responded [09:50:27] 10netops, 10Infrastructure-Foundations, 10SRE: Juniper RA receive bug CVE-2023-28981 - https://phabricator.wikimedia.org/T334916 (10ayounsi) Agreed, the workaround sgtm! [09:52:09] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10MoritzMuehlenhoff) [09:53:34] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10MoritzMuehlenhoff) [09:56:26] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ArielGlenn) [10:04:26] XioNoX: what about ping1003 for today's switch maint, will you redirect ICMP traffic on the routers and shall we just ignore it? [10:08:39] moritzm: that's a good point, not sure what is best [10:09:10] I guess it might be better to temporarily remove the ping redirect on the CRs [10:14:40] sounds good to me, that's also what we do e.g. during reboots of the ping hosts and those last significantly shorter [10:16:23] cool that's a good precident alright, I'll do it before/after [10:17:23] ack! [10:17:27] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10eoghan) [10:23:53] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10MoritzMuehlenhoff) [10:26:41] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10MoritzMuehlenhoff) [10:31:06] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10MoritzMuehlenhoff) [10:39:01] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10jbond) [10:44:38] 10netops, 10Infrastructure-Foundations, 10SRE: Juniper RA receive bug CVE-2023-28981 - https://phabricator.wikimedia.org/T334916 (10Peachey88) [11:51:18] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) jiji@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all active/active services in eqiad: eqiad row D switches upgr... [11:54:30] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10BTullis) [11:58:16] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10BTullis) [11:58:50] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10hnowlan) [12:21:32] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ssingh) [12:25:41] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) jiji@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all active/active services in eqiad: eqiad row D switches upgr... [12:27:34] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) jiji@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all active/active services in eqiad: eqiad row D switches upgr... [12:27:50] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) jiji@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all active/active services in eqiad: eqiad row D switches upgr... [12:39:17] (SystemdUnitFailed) firing: (4) krb5-admin-server.service Failed on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:08:51] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Codfw:row A/B: rack/cable new switches - https://phabricator.wikimedia.org/T332180 (10Papaul) [13:11:32] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7fc7ae6f-d3b2-43ed-b030-194ed6367c80) set by cmooney@cumin1001 for 2:00:00 on 270 hos... [13:12:16] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10cmooney) [13:17:09] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=e714b564-285e-4f22-b860-267d7c23208d) set by cmooney@cumin1001 for 2:00:00 on 1 host(... [13:21:49] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10klausman) [13:28:11] (SystemdUnitFailed) firing: (4) krb5-admin-server.service Failed on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:42:19] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10Marostegui) dbproxy[1016-1017] reloaded [13:42:58] (SystemdUnitFailed) firing: (4) krb5-admin-server.service Failed on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:50:44] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10klausman) [13:52:48] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10MoritzMuehlenhoff) [14:01:27] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10cmooney) [14:12:48] (SystemdUnitFailed) firing: (4) krb5-admin-server.service Failed on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:14:17] (SystemdUnitFailed) firing: (4) krb5-admin-server.service Failed on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:41:56] I'm going through the list of IF refreshes mentioned by Willy [14:42:07] there is sretest2001, which I think was never really in use? [14:42:22] bought in 2017 and still in planned state [14:42:53] since we have two hosts already and don't need sretest* to be redundant, I'll mark that ons as not needing a refresh unless anyone disagrees? [14:47:30] makes sense [14:48:34] unless someone forsees the need for cross dc tests? Perhaps XioNoX or topranks? [14:50:18] ack, but for random special cases, there will also be plenty of codfw insetup:: servers we can briefly use (and then reimage), so it's probably fine [14:50:36] if noone object until tomorrow, I'll mark it as not needing a refresh [14:52:25] +1 [14:54:04] 10netops, 10Infrastructure-Foundations, 10SRE-Sprint-Week-Sustainability-March2023, 10Sustainability (Incident Followup): Cr1-eqiad comms problem when moving to 40G row D handoff - https://phabricator.wikimedia.org/T320566 (10ayounsi) [14:54:17] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ayounsi) [14:55:30] moritzm: I've used sretest2001 for testing when bringing new switches online [14:55:49] i.e. before we move "real" servers to the new switch I've had pa.paul move that one to the rack and test some stuff [14:56:19] but that kind of thing is quite basic, really doesn't need modern hardware so I'm inclined to agree a refresh is not needed [14:56:47] ack [15:08:13] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T3... [15:14:35] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10BTullis) [15:38:43] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T3... [15:39:15] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T3... [15:54:56] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T3... [15:55:12] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T3... [16:00:23] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T3... [16:00:39] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T3... [16:02:44] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T3... [16:04:03] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T3... [16:04:26] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T3... [16:08:24] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T3... [16:08:45] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: End of maintenance - T3... [17:04:03] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10cmooney) 05Open→03Resolved All works complete, no issues to report. [17:04:23] 10netops, 10Infrastructure-Foundations, 10SRE: eqiad/codfw virtual-chassis upgrades - https://phabricator.wikimedia.org/T327248 (10cmooney) [17:04:35] 10netops, 10Infrastructure-Foundations, 10SRE-Sprint-Week-Sustainability-March2023, 10Sustainability (Incident Followup): Cr1-eqiad comms problem when moving to 40G row D handoff - https://phabricator.wikimedia.org/T320566 (10cmooney) [17:32:43] Hello. I'd like to add the perccli utility to the new cephosd* servers to support their HBA. I think that this is the right way to do it, but I'd be glad if anyone could review this please: Thanks. https://gerrit.wikimedia.org/r/c/operations/puppet/+/909707 [17:35:58] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10colewhite) [17:49:59] btullis: ack, I'll have a look tomorrow [18:08:08] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10Jclark-ctr) Verified cables for both Servers below are the ports and cable ids @Papaul cloudswift1001 Rack,C8 U35. port {cloudsw1-... [18:14:17] (SystemdUnitFailed) firing: (4) krb5-admin-server.service Failed on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:30:11] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10Papaul) @Jclark-ctr thanks [19:27:02] 10puppet-compiler, 10Infrastructure-Foundations, 10SRE: PCC failing for an LVS host (false negative) even after manually updating facts - https://phabricator.wikimedia.org/T334680 (10Dzahn) So the code lines where this errors out are: ` $ring_size = $facts['net_driver'][$interface]['driver'] ? {... [19:28:38] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10Papaul) @Jclark-ctr there is no network cable connected to both nodes. [19:29:47] 10puppet-compiler, 10Infrastructure-Foundations, 10SRE: PCC failing for an LVS host (false negative) even after manually updating facts - https://phabricator.wikimedia.org/T334680 (10Dzahn) code was added back in 2018 in https://gerrit.wikimedia.org/r/c/operations/puppet/+/448506 [20:10:01] 10netbox, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Represent sub-interface and bridge device assocations in Netbox - https://phabricator.wikimedia.org/T296832 (10cmooney) @ayounsi has been very helpful with reviewing this patch and it now has a tentative +1 (yay!) In terms of n... [22:14:17] (SystemdUnitFailed) firing: (4) krb5-admin-server.service Failed on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed