[00:24:35] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1018 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [00:24:35] FIRING: [2x] KafkaJumboNoData: Kafka Broker hasn't received data from kafka-jumbo1018 in 60 minutes - https://alerts.wikimedia.org/?q=alertname%3DKafkaJumboNoData [00:28:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [11.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [00:28:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [11.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [00:28:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [11.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [00:33:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [11.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [00:33:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [11.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [00:33:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [11.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [01:05:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [04:24:35] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1018 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [04:24:36] FIRING: [2x] KafkaJumboNoData: Kafka Broker hasn't received data from kafka-jumbo1018 in 60 minutes - https://alerts.wikimedia.org/?q=alertname%3DKafkaJumboNoData [05:48:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb1007.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [05:53:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb1007.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [06:02:03] (03PS1) 10Eileen: Remove space messing up url [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326048 [06:20:23] (03PS1) 10Eileen: Provide gateway_txn_id for Trustly reversal_reversed [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1326049 (https://phabricator.wikimedia.org/T434916) [06:46:38] (03PS1) 10Eileen: Add Vendor ID for StripeMg [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326050 (https://phabricator.wikimedia.org/T432813) [06:54:36] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: INVALID_MESSAGE No payment type found for message [stripe_audit] - https://phabricator.wikimedia.org/T434871#12219119 (10Eileenmcnaughton) We should probably merge into https://phabricator.wikimedia.org/T432813 [07:28:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [07:30:15] (03PS5) 10Eileen: Populate GatewayAccount with the info we have so far.... [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1324509 (https://phabricator.wikimedia.org/T434041) [07:30:58] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1324509 (https://phabricator.wikimedia.org/T434041) (owner: 10Eileen) [07:32:46] (03PS6) 10Eileen: Also put file path on a separate line [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1324513 [07:34:11] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1326052 [08:24:35] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1018 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [08:24:35] FIRING: [2x] KafkaJumboNoData: Kafka Broker hasn't received data from kafka-jumbo1018 in 60 minutes - https://alerts.wikimedia.org/?q=alertname%3DKafkaJumboNoData [08:33:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [12.7] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [08:54:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [6.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [08:54:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [6.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [08:54:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [6.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [08:59:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [6.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [08:59:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [6.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [08:59:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [6.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [09:42:51] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [13.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [12:22:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [12:24:35] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1018 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [12:24:36] FIRING: [2x] KafkaJumboNoData: Kafka Broker hasn't received data from kafka-jumbo1018 in 60 minutes - https://alerts.wikimedia.org/?q=alertname%3DKafkaJumboNoData [12:36:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [12:47:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [43.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [13:42:51] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [40.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [15:48:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [32.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [16:23:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [35.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [16:24:35] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1018 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [16:24:36] FIRING: [2x] KafkaJumboNoData: Kafka Broker hasn't received data from kafka-jumbo1018 in 60 minutes - https://alerts.wikimedia.org/?q=alertname%3DKafkaJumboNoData [16:36:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [59.7] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [18:36:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [19.7] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [18:41:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [14.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [19:11:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [20:24:35] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1018 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [20:24:36] FIRING: [2x] KafkaJumboNoData: Kafka Broker hasn't received data from kafka-jumbo1018 in 60 minutes - https://alerts.wikimedia.org/?q=alertname%3DKafkaJumboNoData [20:27:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [20:32:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [20:37:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [21:17:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [21:22:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [18.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [22:02:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [23:41:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [23:46:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [7.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [23:46:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [7.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [23:46:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [7.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures