[05:13:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [40.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [05:33:36] FIRING: HostTotalProcs: Host high process count (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DHostTotalProcs [05:33:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [47.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [05:38:36] RESOLVED: HostTotalProcs: Host high process count (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DHostTotalProcs [05:48:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb1007.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [05:58:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb1007.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [06:20:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [06:50:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [5.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [06:50:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [5.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [06:50:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [5.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [06:53:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [46.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [06:55:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [5.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [06:55:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [5.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [06:55:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [5.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [06:58:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [45.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [07:35:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [08:23:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [9.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [08:23:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [9.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [08:23:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [9.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [08:28:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [7.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [08:28:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [7.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [08:28:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [7.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [08:33:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [08:36:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [38.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [08:41:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [32.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [08:55:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [55.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [10:35:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [31.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [11:08:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [12.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [11:15:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [150.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [11:35:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [43.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [13:05:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [14:45:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [12.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [14:55:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [150.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [15:36:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [16:36:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [16:45:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [52.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [17:00:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [150.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [18:20:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [18:25:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [18:27:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [19:17:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [20:47:42] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Repeated queue_item insert deadlocks - https://phabricator.wikimedia.org/T436478 (10Lars) 03NEW [20:48:03] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Repeated queue_item insert deadlocks - https://phabricator.wikimedia.org/T436478#12269222 (10Lars) a:03Lars [[ https://github.com/civicrm/civicrm-core/pull/36635 | Upstream PR ]] [21:00:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [150.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [22:05:36] FIRING: HostTotalProcs: Host high process count (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DHostTotalProcs [22:15:36] RESOLVED: HostTotalProcs: Host high process count (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DHostTotalProcs