[00:00:13] RECOVERY - Check systemd state on krb1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:07:39] PROBLEM - Check systemd state on krb1001 is CRITICAL: CRITICAL - degraded: The following units failed: logrotate.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:41:13] PROBLEM - HAProxy HTTPS wikiworkshop.org RSA on cp4037 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 40726 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [00:43:05] RECOVERY - HAProxy HTTPS wikiworkshop.org RSA on cp4037 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 472615 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (RSA) valid until 2023-03-30 14:08:36 +0000 (expires in 25 days) https://wikitech.wikimedia.org/wiki/HTTPS [01:27:36] 10ops-eqiad: Inbound interface errors - https://phabricator.wikimedia.org/T330317 (10phaultfinder) [02:00:25] RECOVERY - Check systemd state on mwlog2002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [02:06:13] PROBLEM - HAProxy HTTPS wikiworkshop.org RSA on cp4044 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 35626 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [02:08:05] RECOVERY - HAProxy HTTPS wikiworkshop.org RSA on cp4044 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 467514 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (RSA) valid until 2023-03-30 14:08:36 +0000 (expires in 25 days) https://wikitech.wikimedia.org/wiki/HTTPS [02:09:45] (JobUnavailable) firing: (2) Reduced availability for job mysql-labs in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:11:11] (03CR) 10Zoranzoki21: [C: 04-1] "Per last comments on task, should be abandoned." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/892955 (https://phabricator.wikimedia.org/T324545) (owner: 10Stang) [02:14:11] Hi, is maybe someone with access of locking accounts on Phabricator maybe around? [02:14:48] Oh, sorry, this is a wrong channel to ask, but I'm hoping that I'll get answer anyways. :) [02:24:45] (JobUnavailable) firing: (2) Reduced availability for job mysql-labs in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:49:11] PROBLEM - HAProxy HTTPS wikiworkshop.org RSA on cp4039 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 33048 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [02:51:03] RECOVERY - HAProxy HTTPS wikiworkshop.org RSA on cp4039 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 464937 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (RSA) valid until 2023-03-30 14:08:36 +0000 (expires in 25 days) https://wikitech.wikimedia.org/wiki/HTTPS [03:01:01] PROBLEM - HAProxy HTTPS wikiworkshop.org ECDSA on cp4041 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [03:02:49] RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp4041 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 295030 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2023-03-30 14:08:29 +0000 (expires in 25 days) https://wikitech.wikimedia.org/wiki/HTTPS [03:24:27] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp4039 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [03:26:19] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp4039 is OK: SSL OK - OCSP staple validity for wikipedia.org has 192820 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2023-05-24 08:07:08 +0000 (expires in 80 days) https://wikitech.wikimedia.org/wiki/HTTPS [03:43:56] Kizule: I have access to othe phabbantool [03:46:55] p858snake: I mean if you could lock this account on Phabricator. https://phabricator.wikimedia.org/p/Onecommaccount/ [03:54:13] Track down the history, done [03:55:27] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp4042 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [03:57:13] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp4042 is OK: SSL OK - OCSP staple validity for wikipedia.org has 190966 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2023-05-24 08:07:08 +0000 (expires in 80 days) https://wikitech.wikimedia.org/wiki/HTTPS [03:58:31] Amazing, thank you p858snake! [04:02:33] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp4037 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [04:04:23] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp4037 is OK: SSL OK - OCSP staple validity for wikipedia.org has 190536 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2023-05-24 08:07:08 +0000 (expires in 80 days) https://wikitech.wikimedia.org/wiki/HTTPS [04:35:45] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp4043 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [04:37:35] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp4043 is OK: SSL OK - OCSP staple validity for wikipedia.org has 188544 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2023-05-24 08:07:08 +0000 (expires in 80 days) https://wikitech.wikimedia.org/wiki/HTTPS [04:47:35] 10ops-eqiad: Inbound interface errors - https://phabricator.wikimedia.org/T330317 (10phaultfinder) [05:13:07] PROBLEM - HAProxy HTTPS wikiworkshop.org ECDSA on cp4039 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [05:14:59] RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp4039 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 287101 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2023-03-30 14:08:29 +0000 (expires in 25 days) https://wikitech.wikimedia.org/wiki/HTTPS [06:12:15] PROBLEM - HAProxy HTTPS wikiworkshop.org ECDSA on cp4039 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [06:13:35] PROBLEM - Backup freshness on backup1001 is CRITICAL: Stale: 1 (gerrit1001), Fresh: 118 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring [06:14:09] RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp4039 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 283551 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2023-03-30 14:08:29 +0000 (expires in 25 days) https://wikitech.wikimedia.org/wiki/HTTPS [06:24:45] (JobUnavailable) firing: Reduced availability for job mysql-labs in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:35:15] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp4043 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [06:38:55] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp4043 is OK: SSL OK - OCSP staple validity for wikipedia.org has 181264 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2023-05-24 08:07:08 +0000 (expires in 80 days) https://wikitech.wikimedia.org/wiki/HTTPS [06:49:55] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp4039 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [06:51:49] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp4039 is OK: SSL OK - OCSP staple validity for wikipedia.org has 180491 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2023-05-24 08:07:08 +0000 (expires in 80 days) https://wikitech.wikimedia.org/wiki/HTTPS [08:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20230305T0800) [08:02:35] 10ops-eqiad: Inbound interface errors - https://phabricator.wikimedia.org/T330317 (10phaultfinder) [09:16:07] RECOVERY - Backup freshness on backup1001 is OK: Fresh: 119 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring [09:22:15] PROBLEM - HAProxy HTTPS wikiworkshop.org RSA on cp4044 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 9464 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [09:24:07] RECOVERY - HAProxy HTTPS wikiworkshop.org RSA on cp4044 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 441352 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (RSA) valid until 2023-03-30 14:08:36 +0000 (expires in 25 days) https://wikitech.wikimedia.org/wiki/HTTPS [10:24:45] (JobUnavailable) firing: Reduced availability for job mysql-labs in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:25:39] PROBLEM - HAProxy HTTPS wikiworkshop.org ECDSA on cp4044 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [10:27:23] RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp4044 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 268356 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2023-03-30 14:08:29 +0000 (expires in 25 days) https://wikitech.wikimedia.org/wiki/HTTPS [10:28:17] PROBLEM - HAProxy HTTPS wikiworkshop.org RSA on cp4039 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 5502 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [10:30:03] RECOVERY - HAProxy HTTPS wikiworkshop.org RSA on cp4039 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 437397 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (RSA) valid until 2023-03-30 14:08:36 +0000 (expires in 25 days) https://wikitech.wikimedia.org/wiki/HTTPS [10:45:35] PROBLEM - HAProxy HTTPS wikiworkshop.org RSA on cp4038 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 4463 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [10:47:25] RECOVERY - HAProxy HTTPS wikiworkshop.org RSA on cp4038 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 436353 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (RSA) valid until 2023-03-30 14:08:36 +0000 (expires in 25 days) https://wikitech.wikimedia.org/wiki/HTTPS [11:12:07] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp4044 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [11:15:49] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp4044 is OK: SSL OK - OCSP staple validity for wikipedia.org has 380651 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2023-05-24 08:07:08 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [11:25:03] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp4044 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [11:26:55] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp4044 is OK: SSL OK - OCSP staple validity for wikipedia.org has 379985 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2023-05-24 08:07:08 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [11:31:23] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp4038 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [11:35:05] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp4038 is OK: SSL OK - OCSP staple validity for wikipedia.org has 379494 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2023-05-24 08:07:08 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [12:31:29] PROBLEM - Check unit status of httpbb_kubernetes_hourly on cumin2002 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [12:32:29] PROBLEM - Check systemd state on cumin2002 is CRITICAL: CRITICAL - degraded: The following units failed: httpbb_kubernetes_hourly.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [13:22:51] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp4043 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [13:24:41] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp4043 is OK: SSL OK - OCSP staple validity for wikipedia.org has 372918 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2023-05-24 08:07:08 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [13:27:33] RECOVERY - Check systemd state on cumin2002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [13:36:23] RECOVERY - Check unit status of httpbb_kubernetes_hourly on cumin2002 is OK: OK: Status of the systemd unit httpbb_kubernetes_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [14:08:16] (03PS7) 10Winston Sung: SiteMatrix config: Add actual (non-deprecated) language code for deprecated language codes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/884494 (https://phabricator.wikimedia.org/T172035) [14:22:34] 10ops-eqiad: Inbound interface errors - https://phabricator.wikimedia.org/T330317 (10phaultfinder) [14:24:45] (JobUnavailable) firing: Reduced availability for job mysql-labs in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:35:13] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp4039 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [14:37:05] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp4039 is OK: SSL OK - OCSP staple validity for wikipedia.org has 368575 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2023-05-24 08:07:08 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:48:29] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp4041 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [14:50:21] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp4041 is OK: SSL OK - OCSP staple validity for wikipedia.org has 367779 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2023-05-24 08:07:08 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:50:21] PROBLEM - HAProxy HTTPS wikipedia.org RSA on cp4042 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [14:52:11] RECOVERY - HAProxy HTTPS wikipedia.org RSA on cp4042 is OK: SSL OK - OCSP staple validity for wikipedia.org has 374869 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (RSA) valid until 2023-05-24 07:09:36 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:08:45] PROBLEM - Kafka MirrorMaker main-codfw_to_main-eqiad max lag in last 10 minutes on alert1001 is CRITICAL: 1.029e+05 gt 1e+05 https://wikitech.wikimedia.org/wiki/Kafka/Administration https://grafana.wikimedia.org/d/000000521/kafka-mirrormaker?var-datasource=eqiad+prometheus/ops&var-lag_datasource=codfw+prometheus/ops&var-mirror_name=main-codfw_to_main-eqiad [15:51:25] PROBLEM - HAProxy HTTPS wikiworkshop.org ECDSA on cp4041 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [15:55:05] RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp4041 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 248694 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2023-03-30 14:08:29 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [16:20:54] (03CR) 10Aklapper: "Can someone please abandon this patch? I lack permissions." [puppet] - 10https://gerrit.wikimedia.org/r/553097 (https://phabricator.wikimedia.org/T238751) (owner: 10Alaa Sarhan) [16:34:59] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp4039 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [16:35:58] (03Abandoned) 10Majavah: Update cron with lb and lb-pool params [puppet] - 10https://gerrit.wikimedia.org/r/553097 (https://phabricator.wikimedia.org/T238751) (owner: 10Alaa Sarhan) [16:36:49] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp4039 is OK: SSL OK - OCSP staple validity for wikipedia.org has 361390 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2023-05-24 08:07:08 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [16:40:47] RECOVERY - Kafka MirrorMaker main-codfw_to_main-eqiad max lag in last 10 minutes on alert1001 is OK: (C)1e+05 gt (W)1e+04 gt 64 https://wikitech.wikimedia.org/wiki/Kafka/Administration https://grafana.wikimedia.org/d/000000521/kafka-mirrormaker?var-datasource=eqiad+prometheus/ops&var-lag_datasource=codfw+prometheus/ops&var-mirror_name=main-codfw_to_main-eqiad [17:08:37] PROBLEM - HAProxy HTTPS wikiworkshop.org RSA on cp4037 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [17:12:19] RECOVERY - HAProxy HTTPS wikiworkshop.org RSA on cp4037 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 413260 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (RSA) valid until 2023-03-30 14:08:36 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [17:17:31] PROBLEM - HAProxy HTTPS wikipedia.org RSA on cp4043 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [17:19:21] RECOVERY - HAProxy HTTPS wikipedia.org RSA on cp4043 is OK: SSL OK - OCSP staple validity for wikipedia.org has 366038 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (RSA) valid until 2023-05-24 07:09:36 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [17:42:35] 10ops-eqiad: Inbound interface errors - https://phabricator.wikimedia.org/T330317 (10phaultfinder) [17:48:07] PROBLEM - HAProxy HTTPS wikipedia.org RSA on cp4041 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [17:49:57] RECOVERY - HAProxy HTTPS wikipedia.org RSA on cp4041 is OK: SSL OK - OCSP staple validity for wikipedia.org has 364202 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (RSA) valid until 2023-05-24 07:09:36 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:00:39] (03CR) 10Aklapper: "Brennen: How to proceed?" [puppet] - 10https://gerrit.wikimedia.org/r/877188 (https://phabricator.wikimedia.org/T155130) (owner: 10Aklapper) [18:24:45] (JobUnavailable) firing: Reduced availability for job mysql-labs in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:27:51] PROBLEM - HAProxy HTTPS wikiworkshop.org ECDSA on cp4037 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [18:29:43] RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp4037 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 239417 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2023-03-30 14:08:29 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:50:35] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp4044 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [18:52:25] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp4044 is OK: SSL OK - OCSP staple validity for wikipedia.org has 353253 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2023-05-24 08:07:08 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [19:01:33] PROBLEM - restbase endpoints health on restbase1029 is CRITICAL: /en.wikipedia.org/v1/page/talk/{title} (Get structured talk page for enwiki Salt article) is CRITICAL: Test Get structured talk page for enwiki Salt article returned the unexpected status 503 (expecting: 200) https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [19:03:25] RECOVERY - restbase endpoints health on restbase1029 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [20:07:34] 10ops-eqiad: Inbound interface errors - https://phabricator.wikimedia.org/T330317 (10phaultfinder) [20:08:47] PROBLEM - HAProxy HTTPS wikipedia.org RSA on cp4042 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [20:10:33] RECOVERY - HAProxy HTTPS wikipedia.org RSA on cp4042 is OK: SSL OK - OCSP staple validity for wikipedia.org has 355767 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (RSA) valid until 2023-05-24 07:09:36 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [20:11:13] PROBLEM - HAProxy HTTPS wikipedia.org RSA on cp4041 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [20:12:59] RECOVERY - HAProxy HTTPS wikipedia.org RSA on cp4041 is OK: SSL OK - OCSP staple validity for wikipedia.org has 355620 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (RSA) valid until 2023-05-24 07:09:36 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [20:38:01] PROBLEM - HAProxy HTTPS wikiworkshop.org RSA on cp4041 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [20:41:43] RECOVERY - HAProxy HTTPS wikiworkshop.org RSA on cp4041 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 400697 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (RSA) valid until 2023-03-30 14:08:36 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [20:47:59] PROBLEM - HAProxy HTTPS wikipedia.org RSA on cp4052 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [20:49:49] RECOVERY - HAProxy HTTPS wikipedia.org RSA on cp4052 is OK: SSL OK - OCSP staple validity for wikipedia.org has 353410 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (RSA) valid until 2023-05-24 07:09:36 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [21:02:47] PROBLEM - HAProxy HTTPS wikiworkshop.org ECDSA on cp4040 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [21:06:29] RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp4040 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 230010 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2023-03-30 14:08:29 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [21:10:11] PROBLEM - HAProxy HTTPS wikiworkshop.org RSA on cp4044 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [21:12:03] RECOVERY - HAProxy HTTPS wikiworkshop.org RSA on cp4044 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 398876 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (RSA) valid until 2023-03-30 14:08:36 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [21:19:48] (ProbeDown) firing: (2) Service centrallog1001:6514 has failed probes (tcp_rsyslog_receiver_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#centrallog1001:6514 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:21:10] (ProbeDown) resolved: (2) Service centrallog1001:6514 has failed probes (tcp_rsyslog_receiver_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#centrallog1001:6514 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:28:09] PROBLEM - HAProxy HTTPS wikipedia.org RSA on cp4044 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [21:28:55] PROBLEM - HAProxy HTTPS wikipedia.org RSA on cp4037 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [21:29:59] RECOVERY - HAProxy HTTPS wikipedia.org RSA on cp4044 is OK: SSL OK - OCSP staple validity for wikipedia.org has 351000 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (RSA) valid until 2023-05-24 07:09:36 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [21:30:47] RECOVERY - HAProxy HTTPS wikipedia.org RSA on cp4037 is OK: SSL OK - OCSP staple validity for wikipedia.org has 350952 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (RSA) valid until 2023-05-24 07:09:36 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [22:09:17] PROBLEM - HAProxy HTTPS wikipedia.org RSA on cp4041 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [22:09:27] PROBLEM - HAProxy HTTPS wikipedia.org RSA on cp4037 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [22:11:09] RECOVERY - HAProxy HTTPS wikipedia.org RSA on cp4041 is OK: SSL OK - OCSP staple validity for wikipedia.org has 348531 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (RSA) valid until 2023-05-24 07:09:36 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [22:11:19] RECOVERY - HAProxy HTTPS wikipedia.org RSA on cp4037 is OK: SSL OK - OCSP staple validity for wikipedia.org has 348521 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (RSA) valid until 2023-05-24 07:09:36 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [22:12:59] PROBLEM - HAProxy HTTPS wikiworkshop.org RSA on cp4042 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [22:14:49] RECOVERY - HAProxy HTTPS wikiworkshop.org RSA on cp4042 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 395110 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (RSA) valid until 2023-03-30 14:08:36 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [22:24:45] (JobUnavailable) firing: Reduced availability for job mysql-labs in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [22:47:40] (03PS1) 10Raymond Ndibe: wmcs: nfs: primary: introduce missing hiera keys for maintain_dbusers [puppet] - 10https://gerrit.wikimedia.org/r/894225 (https://phabricator.wikimedia.org/T303663) [23:16:07] (03PS1) 10Raymond Ndibe: wmcs:nfs:replica_cnf_api_service: update PAWS_REPLICA_CNF_PATH [puppet] - 10https://gerrit.wikimedia.org/r/894227 (https://phabricator.wikimedia.org/T303663) [23:20:17] PROBLEM - HAProxy HTTPS wikipedia.org RSA on cp4038 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [23:20:35] PROBLEM - HAProxy HTTPS wikiworkshop.org RSA on cp4041 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [23:22:09] RECOVERY - HAProxy HTTPS wikipedia.org RSA on cp4038 is OK: SSL OK - OCSP staple validity for wikipedia.org has 344271 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (RSA) valid until 2023-05-24 07:09:36 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [23:22:25] RECOVERY - HAProxy HTTPS wikiworkshop.org RSA on cp4041 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 391053 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (RSA) valid until 2023-03-30 14:08:36 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [23:37:59] PROBLEM - HAProxy HTTPS wikipedia.org RSA on cp4040 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:did not receive the required stapled OCSP response https://wikitech.wikimedia.org/wiki/HTTPS [23:39:49] RECOVERY - HAProxy HTTPS wikipedia.org RSA on cp4040 is OK: SSL OK - OCSP staple validity for wikipedia.org has 343210 seconds left:Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (RSA) valid until 2023-05-24 07:09:36 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS