[00:00:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [36.7] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [00:04:26] (03PS1) 10Lars SG: Fix Matching Gifts LYBUNT report [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1327663 (https://phabricator.wikimedia.org/T428474) [00:10:01] (03CR) 10Eileen: [C:03+2] Fix Matching Gifts LYBUNT report [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1327663 (https://phabricator.wikimedia.org/T428474) (owner: 10Lars SG) [00:25:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [150.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [00:28:19] thanks eileen: would you be able to take a look at these two https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1326407 [00:28:27] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1325561 [00:35:15] (03Merged) 10jenkins-bot: Fix Matching Gifts LYBUNT report [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1327663 (https://phabricator.wikimedia.org/T428474) (owner: 10Lars SG) [01:15:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [150.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [02:28:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [150.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [02:48:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [70.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [02:55:05] (03PS1) 10Dmiranda: Identify moto transactions and keep their metadata [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1327676 (https://phabricator.wikimedia.org/T432672) [03:10:42] (03PS2) 10Dmiranda: WIP: Identify moto transactions and keep their metadata [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1327676 (https://phabricator.wikimedia.org/T432672) [04:49:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [62.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [05:17:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [05:49:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [30.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [06:07:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [123.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [06:22:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [31.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [06:43:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [39.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [06:57:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [13.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [06:58:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [35.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [07:22:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [43.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [07:47:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [50.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [08:45:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [64.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [08:45:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [14.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [09:18:51] FIRING: [4x] SwitchCoreInterfaceDown: Switch core interface down - fasw2-e15a-eqiad:et-0/0/47 (Core: pfw1-eqiad:et-0/1/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [09:19:11] FIRING: PfwCoreBGPDown: Fundraising Firewall core BGP session down between pfw1-codfw and (null) (10.195.0.248) - group VPN - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=pfw1-codfw:9804&var-bgp_group=VPN&var-bgp_neighbor=(null) - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [09:20:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [64.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [09:20:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [124.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [09:24:40] FIRING: WidespreadPuppetFailure: Puppet has failed on over 5% of hosts - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [09:24:41] FIRING: [9x] NodeDown: Node fran1002 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [09:24:41] FIRING: [6x] NodeDown: Node civi2002 has been down for long. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [09:24:56] FIRING: [6x] NodeDown: Node fransc1001 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [09:25:09] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [64.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [09:25:09] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [129.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [09:25:17] FIRING: [6x] HAProxyExporterDown: HAProxy on fransc2001.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [09:25:17] FIRING: [8x] HAProxyExporterDown: HAProxy on fransc2001.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [09:25:32] FIRING: [6x] HAProxyExporterDown: HAProxy on fransc1001.frack.eqiad.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [09:25:32] RESOLVED: [4x] HAProxyExporterDown: HAProxy on fransc1001.frack.eqiad.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [09:28:51] RESOLVED: [4x] SwitchCoreInterfaceDown: Switch core interface down - fasw2-e15a-eqiad:et-0/0/47 (Core: pfw1-eqiad:et-0/1/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [09:29:11] RESOLVED: PfwCoreBGPDown: Fundraising Firewall core BGP session down between pfw1-codfw and (null) (10.195.0.248) - group VPN - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=pfw1-codfw:9804&var-bgp_group=VPN&var-bgp_neighbor=(null) - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [09:29:41] RESOLVED: [62x] NodeDown: Node civi1002 has been down for long. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [09:29:56] RESOLVED: [62x] NodeDown: Node civi1002 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [09:30:32] RESOLVED: [12x] HAProxyExporterDown: HAProxy on fransc1001.frack.eqiad.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [09:31:33] FIRING: RsyslogBacklog: 3 ryslog files queued up on frlog2002.frack.codfw.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DRsyslogBacklog [09:34:40] RESOLVED: WidespreadPuppetFailure: Puppet has failed on over 5% of hosts - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [09:36:33] RESOLVED: RsyslogBacklog: 3 ryslog files queued up on frlog2002.frack.codfw.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DRsyslogBacklog [09:41:05] FIRING: KafkaBrokerDown: Kafka Broker kafka-jumbo1018 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [09:50:09] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [70.7] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [09:51:05] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1018 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [10:04:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [5.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [10:04:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [5.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [10:04:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [5.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [10:06:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [63.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [10:09:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [5.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [10:09:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [5.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [10:09:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [5.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [10:12:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [9.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [10:12:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [9.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [10:12:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [9.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [10:22:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [5.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [10:22:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [5.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [10:22:43] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [5.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [10:39:26] (03PS6) 10Wfan: Add venmo to combo wiki [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1327174 (https://phabricator.wikimedia.org/T432231) [10:39:26] (03PS10) 10Wfan: Add post recurring convert popup modal [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326359 (https://phabricator.wikimedia.org/T433143) [10:44:21] (03PS1) 10Wfan: Add a translation for google Pay [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1328161 [11:20:09] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [13.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [11:21:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [37.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [11:54:22] (03PS1) 10Ladsgroup: Remove calls to Database::getCacheSetOptions() [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1328173 (https://phabricator.wikimedia.org/T424357) [12:14:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [16.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [12:14:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [16.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [12:14:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [16.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [12:19:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [6.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [12:19:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [6.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [12:19:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [6.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [12:24:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [61.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [12:29:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [44.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [12:45:11] 06Fundraising-Analysis, 06Fundraising-Backlog, 06Data-Platform-SRE: Upgrade airflow-fr-tech to 3.3.x - https://phabricator.wikimedia.org/T435479#12241221 (10atsuko) [13:29:35] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Tech onboarding task: Minor UI bug in CiviCRM: counter of recurring contributions is wrong in sidebar - https://phabricator.wikimedia.org/T435613 (10Laurabarluzzi) 03NEW [13:31:49] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Tech onboarding task: Minor UI bug in CiviCRM: counter of recurring contributions is wrong in sidebar - https://phabricator.wikimedia.org/T435613#12241335 (10Laurabarluzzi) [13:51:05] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1018 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [13:52:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [101.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [13:52:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [14.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [13:57:29] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-failmail: [ACH ]Validation error: Payment method has not been successfully stored - https://phabricator.wikimedia.org/T435615 (10Laurabarluzzi) 03NEW [13:57:44] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-failmail: [ACH] Validation error: Payment method has not been successfully stored - https://phabricator.wikimedia.org/T435615#12241400 (10Laurabarluzzi) [13:59:02] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Tech onboarding task: Minor UI bug in CiviCRM: counter of recurring contributions is wrong in sidebar - https://phabricator.wikimedia.org/T435613#12241410 (10Lars) @cstone Do you have a sense if people are actuall... [14:02:09] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-failmail: [ACH] Validation error: Payment method has not been successfully stored - https://phabricator.wikimedia.org/T435615#12241424 (10Laurabarluzzi) [14:12:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [40.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [14:54:55] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [13.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [15:06:45] 06Fundraising-Analysis, 06Fundraising-Backlog, 06Data-Platform-SRE: Upgrade airflow-fr-tech to 3.3.x - https://phabricator.wikimedia.org/T435479#12241724 (10atsuko) @AStein-WMF it should now run past this error, I can't fully test it because I have no permissions to your HDFS folder. [15:53:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [52.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [15:58:28] 03Fundraising Sprint: Ottoman-o-war, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint: Remove un-needed separate endowment/foundation calculated fields from Civi & Acoustic - https://phabricator.wikimedia.org/T418885#12241945 (10Lars) Added [[ https://wikitech.wikimedia.org/wiki/... [15:58:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [50.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [16:08:23] (03CR) 10Wfan: Add post recurring convert popup modal (032 comments) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326359 (https://phabricator.wikimedia.org/T433143) (owner: 10Wfan) [16:24:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [16:29:05] (03PS7) 10Wfan: Add venmo to combo wiki [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1327174 (https://phabricator.wikimedia.org/T432231) [16:33:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [54.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [16:54:33] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donation-Processing: Chariot data issue - DAFpay payment method - https://phabricator.wikimedia.org/T434317#12242165 (10MDemosWMF) These have been cleaned up in Civi. [16:59:44] (03PS11) 10Wfan: Add post recurring convert popup modal [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326359 (https://phabricator.wikimedia.org/T433143) [17:00:38] (03PS2) 10Wfan: Add a translation for google Pay [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1328161 [17:13:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [31.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [17:14:34] (03CR) 10Dmiranda: [C:03+2] "Thanks @wfan@wikimedia.org!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1328161 (owner: 10Wfan) [17:22:32] 06Fundraising-Backlog, 10FR-Donation-Processing: Duplicate banking institution soft credit - https://phabricator.wikimedia.org/T435643 (10MDemosWMF) 03NEW [17:50:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [59.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [17:53:30] (03CR) 10Dmiranda: WIP: Identify moto transactions and keep their metadata (031 comment) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1327676 (https://phabricator.wikimedia.org/T432672) (owner: 10Dmiranda) [17:53:55] (03PS3) 10Dmiranda: Identify moto transactions and keep their metadata [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1327676 (https://phabricator.wikimedia.org/T432672) [17:54:56] FIRING: [2x] KafkaBrokerDown: Kafka Broker kafka-jumbo1018 is down: [0] - https://alerts.wikimedia.org/?q=alertname%3DKafkaBrokerDown [18:15:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [36.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [18:40:51] 06Fundraising-Analysis, 06Fundraising-Backlog, 06Data-Platform-SRE: Upgrade airflow-fr-tech to 3.3.x - https://phabricator.wikimedia.org/T435479#12242496 (10AStein-WMF) Looks like it's still not transferring. after 10 minutes it requeues twice and then fails. lmk what oth er info would be helpful to provide.... [18:43:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [75.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [18:48:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [75.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [18:49:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [19:07:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [33.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [19:12:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [33.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [19:13:44] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE: Link and open tracking for CiviMail show as disabled when creating a new mailing, but they are not - https://phabricator.wikimedia.org/T435386#12242536 (10Lars) a:03Lars [19:28:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [53.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [19:43:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [51.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [20:05:05] (03PS5) 10Jdlrobson: startUp: Suppress banners for users in prohibited experiments [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1327214 (https://phabricator.wikimedia.org/T435658) [20:19:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [32.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [20:24:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [32.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [20:26:27] 06Fundraising-Analysis, 06Fundraising-Backlog, 06Data-Platform-SRE: Upgrade airflow-fr-tech to 3.3.x - https://phabricator.wikimedia.org/T435479#12242708 (10amastilovic) I've played with Airflow 3 branch in my devenv and here are my findings so far: 1. `ExternalTaskMarker` in Airflow 3 removed the `executio... [20:36:34] (03CR) 10Elliottetzkorn: [C:03+1] "Great thanks for addressing the comments" [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1327214 (https://phabricator.wikimedia.org/T435658) (owner: 10Jdlrobson) [20:39:13] (03CR) 10Kimberly Sarabia: [C:03+2] startUp: Suppress banners for users in prohibited experiments [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1327214 (https://phabricator.wikimedia.org/T435658) (owner: 10Jdlrobson) [20:43:03] (03Merged) 10jenkins-bot: startUp: Suppress banners for users in prohibited experiments [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1327214 (https://phabricator.wikimedia.org/T435658) (owner: 10Jdlrobson) [21:01:04] (03PS1) 10Laurabarluzzi: Add testPatternFilterIpCountryMatch to test usage of ip_country in fraud filters [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1328242 [21:06:27] (03PS2) 10Laurabarluzzi: Add testPatternFilterIpCountryMatch to test usage of ip_country in fraud filters [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1328242 [21:11:56] (03PS2) 10Krinkle: Remove calls to Database::getCacheSetOptions() [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1328173 (https://phabricator.wikimedia.org/T424357) (owner: 10Ladsgroup) [21:12:00] (03CR) 10Krinkle: [C:03+2] Remove calls to Database::getCacheSetOptions() [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1328173 (https://phabricator.wikimedia.org/T424357) (owner: 10Ladsgroup) [21:12:23] (03PS2) 10Krinkle: Remove calls to Database::getCacheSetOptions() [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1325946 (https://phabricator.wikimedia.org/T424357) (owner: 10Ladsgroup) [21:14:43] (03CR) 10CI reject: [V:04-1] Remove calls to Database::getCacheSetOptions() [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1325946 (https://phabricator.wikimedia.org/T424357) (owner: 10Ladsgroup) [21:16:27] (03Merged) 10jenkins-bot: Remove calls to Database::getCacheSetOptions() [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1328173 (https://phabricator.wikimedia.org/T424357) (owner: 10Ladsgroup) [23:03:11] FIRING: NodeDown: Node fran1002 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fran1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [23:08:11] RESOLVED: NodeDown: Node fran1002 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fran1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [23:38:30] (03PS1) 10Lars SG: Add optIn url param to civiproxy for double opt in [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1328254 (https://phabricator.wikimedia.org/T434150) [23:47:09] 03Fundraising Sprint: Ottoman-o-war, 06Fundraising-Backlog, 10fundraising-tech-ops, 03Civi RoadMap FY 26-27: FR-Tech FY2627Q1 maintenance window - https://phabricator.wikimedia.org/T419557#12243291 (10Dwisehaupt) [23:56:25] (03CR) 10Cstone: [C:03+2] "Looks good, naming is hard we probably want to name the component like Redirect or something but we can change that later" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1327174 (https://phabricator.wikimedia.org/T432231) (owner: 10Wfan) [23:57:03] Thanks Christine~ [23:58:10] (03Merged) 10jenkins-bot: Add venmo to combo wiki [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1327174 (https://phabricator.wikimedia.org/T432231) (owner: 10Wfan)