[00:39:31] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/931924 [00:39:37] (03CR) 10TrainBranchBot: [C: 03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/931924 (owner: 10TrainBranchBot) [00:59:25] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/931924 (owner: 10TrainBranchBot) [01:30:18] !log andrew@deploy1002 Started deploy [horizon/deploy@9d02cd6]: asdf [01:30:19] !log andrew@deploy1002 deploy aborted: asdf (duration: 00m 01s) [01:31:01] !log andrew@deploy1002 Started deploy [horizon/deploy@9d02cd6]: installing (but not registering) magnum-ui [01:35:07] !log andrew@deploy1002 Finished deploy [horizon/deploy@9d02cd6]: installing (but not registering) magnum-ui (duration: 04m 05s) [01:45:42] !log andrew@deploy1002 Started deploy [horizon/deploy@9d02cd6]: installing (but not registering) magnum-ui [02:07:35] (JobUnavailable) firing: (4) Reduced availability for job cloud_dev_pdns in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:32:35] (JobUnavailable) firing: (4) Reduced availability for job cloud_dev_pdns in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:54:16] (MediaWikiLatencyExceeded) firing: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus/ops&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [02:59:16] (MediaWikiLatencyExceeded) resolved: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus/ops&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [03:20:57] 10SRE, 10Wikimedia-Mailing-lists: Add custom footer linking to Privacy Policy in Postorious and Hyperkitty - https://phabricator.wikimedia.org/T340375 (10EpicPupper) Mailing lists are not only used for technical things so the UCoC should probably be linked. Also, the non-wiki privacy policy https://foundation.... [03:22:52] 10SRE, 10Wikimedia-Mailing-lists: Add custom footer linking to Privacy Policy in Postorious and Hyperkitty - https://phabricator.wikimedia.org/T340375 (10Peachey88) > Additionally, I don't see the CoC linked in Gitiles? I wonder if it could be added. This is out of scope for this task, Please avoid task creep... [03:35:19] (ProbeDown) firing: (6) Service ml-cache1001:7001 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:42:59] PROBLEM - mailman archives on lists1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [03:43:01] PROBLEM - mailman list info on lists1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [03:44:27] RECOVERY - mailman archives on lists1001 is OK: HTTP OK: HTTP/1.1 200 OK - 50136 bytes in 4.670 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [03:44:29] RECOVERY - mailman list info on lists1001 is OK: HTTP OK: HTTP/1.1 200 OK - 8572 bytes in 3.994 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [03:54:53] RECOVERY - Router interfaces on cr3-esams is OK: OK: host 91.198.174.245, interfaces up: 84, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [03:55:49] RECOVERY - Router interfaces on cr2-drmrs is OK: OK: host 185.15.58.129, interfaces up: 62, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [04:00:29] RECOVERY - Check systemd state on build2001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [06:20:19] (ProbeDown) firing: (7) Service centrallog2002:6514 has failed probes (tcp_rsyslog_receiver_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [06:31:16] (MediaWikiLatencyExceeded) firing: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus/ops&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [06:32:35] (JobUnavailable) firing: (2) Reduced availability for job cloud_dev_pdns in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:36:16] (MediaWikiLatencyExceeded) resolved: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus/ops&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [07:00:04] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20230625T0700) [07:47:01] (03PS1) 10Elukey: role::ml_cache::storage: add Prometheus control plane nodes to ferm [puppet] - 10https://gerrit.wikimedia.org/r/932657 [07:47:52] (03PS2) 10Elukey: role::ml_cache::storage: add Prometheus control plane nodes to ferm [puppet] - 10https://gerrit.wikimedia.org/r/932657 [07:49:51] (03CR) 10Elukey: [C: 03+2] role::ml_cache::storage: add Prometheus control plane nodes to ferm [puppet] - 10https://gerrit.wikimedia.org/r/932657 (owner: 10Elukey) [07:51:12] /9 [08:33:25] 10SRE, 10Wikimedia-Mailing-lists: Create a new mailing list - https://phabricator.wikimedia.org/T340380 (10Sai10ukazuki) [08:44:11] 10SRE, 10Wikimedia-Mailing-lists: Create wikija-g mailing list - https://phabricator.wikimedia.org/T340380 (10Peachey88) [10:01:17] PROBLEM - Host parse1012 is DOWN: PING CRITICAL - Packet loss = 100% [10:02:19] RECOVERY - Host parse1012 is UP: PING OK - Packet loss = 0%, RTA = 3.16 ms [10:02:25] 10Puppet, 10Infrastructure-Foundations, 10Project-Admins, 10PM: Clarify Puppet tag - https://phabricator.wikimedia.org/T295221 (10Aklapper) 05Open→03Resolved I guess we're done then. Thanks! [10:04:16] (MediaWikiLatencyExceeded) firing: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus/ops&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [10:09:16] (MediaWikiLatencyExceeded) resolved: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus/ops&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [10:20:19] (ProbeDown) firing: (6) Service ml-cache1001:7001 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:29:18] (KubernetesAPILatency) firing: High Kubernetes API latency (LIST secrets) on k8s-staging@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/000000435?var-site=eqiad&var-cluster=k8s-staging - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [10:32:35] (JobUnavailable) firing: (2) Reduced availability for job cloud_dev_pdns in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:34:18] (KubernetesAPILatency) resolved: High Kubernetes API latency (LIST secrets) on k8s-staging@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/000000435?var-site=eqiad&var-cluster=k8s-staging - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [13:30:33] PROBLEM - Host parse1012 is DOWN: PING CRITICAL - Packet loss = 100% [13:31:11] RECOVERY - Host parse1012 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [14:06:41] PROBLEM - WDQS SPARQL on wdqs1015 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [14:07:35] (JobUnavailable) firing: (4) Reduced availability for job cloud_dev_pdns in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:11:19] RECOVERY - WDQS SPARQL on wdqs1015 is OK: HTTP OK: HTTP/1.1 200 OK - 692 bytes in 7.955 second response time https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [14:17:35] (JobUnavailable) firing: (4) Reduced availability for job cloud_dev_pdns in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:20:19] (ProbeDown) firing: (6) Service ml-cache1001:7001 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:40:57] (03PS1) 10Elukey: Revert "role::ml_cache::storage: add Prometheus control plane nodes to ferm" [puppet] - 10https://gerrit.wikimedia.org/r/932639 [14:41:23] (03CR) 10Elukey: [C: 03+2] Revert "role::ml_cache::storage: add Prometheus control plane nodes to ferm" [puppet] - 10https://gerrit.wikimedia.org/r/932639 (owner: 10Elukey) [14:42:37] (03PS3) 10Elukey: cassandra::instance::monitoring: move alerts to prometheus [puppet] - 10https://gerrit.wikimedia.org/r/932427 (https://phabricator.wikimedia.org/T288470) [14:42:39] (03PS1) 10Elukey: profile::cassandra: allow Prometheus nodes to check ports [puppet] - 10https://gerrit.wikimedia.org/r/932663 [14:44:13] (03CR) 10Elukey: [V: 03+1] "PCC SUCCESS (CORE_DIFF 3): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/41992/console" [puppet] - 10https://gerrit.wikimedia.org/r/932663 (owner: 10Elukey) [14:52:51] (03CR) 10Majavah: [C: 04-1] "The base firewall rules should already permit all traffic from Prometheus nodes, is that not working?" [puppet] - 10https://gerrit.wikimedia.org/r/932663 (owner: 10Elukey) [15:40:29] (03CR) 10Ayounsi: [C: 03+1] Probenet: Restore mapping for Nigeria [dns] - 10https://gerrit.wikimedia.org/r/932468 (https://phabricator.wikimedia.org/T337318) (owner: 10Jameel Kaisar) [16:57:03] PROBLEM - Host parse1012 is DOWN: PING CRITICAL - Packet loss = 100% [16:58:03] RECOVERY - Host parse1012 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [17:52:20] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10suffusion_of_yellow) Ok, WTF: ` (await (new mw.ForeignApi('https://test.wikipedia.org/w/api.php')).get({action: "abusefilterevalexpression", expressi... [17:59:09] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10Daimona) >>! In T340068#8962684, @suffusion_of_yellow wrote: > Ok, WTF: > > ` > (await (new mw.ForeignApi('https://test.wikipedia.org/w/api.php')).ge... [18:11:01] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10Reedy) I don't know if it's relevant... But test.wikipedia.org is now on k8s ({T337489}) so may have some newer packages somewhere (I don't know what... [18:12:49] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10Daimona) >>! In T340068#8962697, @Reedy wrote: > I don't know if it's relevant... But test.wikipedia.org is now on k8s ({T337489}) so may have some ne... [18:13:45] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10Reedy) >>! In T340068#8962699, @Daimona wrote: >>>! In T340068#8962697, @Reedy wrote: >> I don't know if it's relevant... But test.wikipedia.org is no... [18:16:23] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10Daimona) >>! In T340068#8962700, @Reedy wrote: >>>! In T340068#8962699, @Daimona wrote: >>>>! In T340068#8962697, @Reedy wrote: >>> I don't know if it... [18:17:35] (JobUnavailable) firing: (2) Reduced availability for job cloud_dev_pdns in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:19:15] 10ops-eqiad: ManagementSSHDown - https://phabricator.wikimedia.org/T340400 (10phaultfinder) [18:20:19] (ProbeDown) firing: (6) Service ml-cache1001:7001 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:23:54] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10suffusion_of_yellow) Still getting this for GET requests, but not POST requests on enwiki, even when I throw in junk to prevent caching: ` (async ()... [18:27:26] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10suffusion_of_yellow) But if I prefix the regex with `(*NO_JIT)` I get the correct result 100% of the time. [18:27:34] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10Daimona) >>! In T340068#8962707, @suffusion_of_yellow wrote: > Still getting this for GET requests, but not POST requests on enwiki, even when I throw... [18:32:00] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10Daimona) >>! In T340068#8962709, @Daimona wrote: > ` > 'enwiki' > {"post":[43,0],"get":[53,4]} > ` But also: ` 'itwiki' {"post":[45,1],"get":[53,1]... [18:35:50] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10Daimona) And also: ` daimona@mwmaint1002:~$ mwscript shell.php --wiki=itwiki Psy Shell v0.11.10 (PHP 7.4.33 — cli) by Justin Hileman > preg_match("/... [19:10:09] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10suffusion_of_yellow) From https://github.com/php/php-src/blob/PHP-7.3.31/ext/pcre/php_pcre.c: ` #ifdef HAVE_PCRE_JIT_SUPPORT if (PCRE_G(jit)) { /*... [19:12:50] (03CR) 10Paladox: "This change is ready for review." [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/932640 (owner: 10Paladox) [19:31:37] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10taavi) Wikimedia sites include the app server name in the `server` response header - if you suspect there's some per-server differences the values of... [19:40:38] (03CR) 10Paladox: "This change is ready for review." [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/932641 (https://phabricator.wikimedia.org/T340372) (owner: 10Paladox) [19:40:58] (03CR) 10CI reject: [V: 04-1] Fix wm-custom-links to show links in footer again [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/932641 (https://phabricator.wikimedia.org/T340372) (owner: 10Paladox) [19:41:31] (03PS3) 10Paladox: Fix wm-custom-links to show links in footer again [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/932641 (https://phabricator.wikimedia.org/T340372) [19:50:22] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10suffusion_of_yellow) ` (async () => { let r = {}; for(let x = 0; x < 1000; x++ ) { let method = Math.random() < 0.5 ? "post" : "get"; l... [19:59:17] PROBLEM - Host parse1012 is DOWN: PING CRITICAL - Packet loss = 100% [20:00:03] RECOVERY - Host parse1012 is UP: PING OK - Packet loss = 0%, RTA = 1.86 ms [21:02:01] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10Daimona) It seems not to depend on the server... And apparently it's not even deterministic... One potentially useful thing could be to verify if the... [21:03:56] (03PS3) 10Msz2001: Update plwiki autopromote per consensus [mediawiki-config] - 10https://gerrit.wikimedia.org/r/932642 (https://phabricator.wikimedia.org/T340397) [21:10:39] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10suffusion_of_yellow) When are y'all upgrading to Debian 11? That has PCRE version 10.36, and I cannot even reproduce this with 10.33. But then again... [21:58:49] (03CR) 10Bking: [C: 03+1] query_service: align all hiera configuration to the same order (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/930191 (owner: 10Gehel) [22:17:35] (JobUnavailable) firing: (2) Reduced availability for job cloud_dev_pdns in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [22:20:19] (ProbeDown) firing: (6) Service ml-cache1001:7001 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:19:55] PROBLEM - Host parse1012 is DOWN: PING CRITICAL - Packet loss = 100% [23:20:43] RECOVERY - Host parse1012 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [23:23:11] PROBLEM - mailman archives on lists1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [23:23:39] PROBLEM - mailman list info on lists1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [23:26:07] RECOVERY - mailman archives on lists1001 is OK: HTTP OK: HTTP/1.1 200 OK - 50134 bytes in 0.063 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [23:26:33] RECOVERY - mailman list info on lists1001 is OK: HTTP OK: HTTP/1.1 200 OK - 8571 bytes in 0.286 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [23:41:51] 10SRE, 10AbuseFilter, 10serviceops, 10PHP 7.4 support: Regular expression "х[ÿý]и" match "х и" in Abusefilter - https://phabricator.wikimedia.org/T340068 (10Reedy) {T291916}