[01:00:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [150.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [01:48:46] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_46) - 10https://gerrit.wikimedia.org/r/1331018 (owner: 10L10n-bot) [04:51:56] (03PS1) 10Lars SG: Improve deadlock handling in QueueConsumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332533 (https://phabricator.wikimedia.org/T436157) [04:52:19] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Donation queue consumer fails due to deadlocks - https://phabricator.wikimedia.org/T436157#12269552 (10Lars) a:03Lars [04:55:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [49.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [05:02:43] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Donation queue consumer fails due to deadlocks - https://phabricator.wikimedia.org/T436157#12269555 (10Lars) The patch above improves the deadlocks retry / throw for this specific problem. But Eile... [06:58:07] 14Fundraising Sprint: Kondo-approved bare room: Sprint L Prospects - https://phabricator.wikimedia.org/T430364#12269686 (10Aklapper) @SBurnett-WMF / @greg: The project tag on this open task got archived; please add an active project tag so this task can be found on an active workboard, or resolve this task - tha... [08:00:36] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-failmail: process-control lead_generation_queue_consume : handle Invalid address Exception - https://phabricator.wikimedia.org/T436215#12269919 (10Laurabarluzzi) [08:58:51] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-failmail: Sudden invalid_payment_method - https://phabricator.wikimedia.org/T436503 (10Laurabarluzzi) 03NEW [09:09:28] 03Fundraising Sprint: Pedestal to the Metal, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint, 13Patch-For-Review: Show iframe with donor details in Zendesk - https://phabricator.wikimedia.org/T433866#12270128 (10krobinson) Thanks @Lars //Other data we might want to add (easi... [10:29:40] (03PS4) 10Laurabarluzzi: Add missing valid ISO countries to Gravy to stop PayPal falling back to paypal_ec [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1331812 (https://phabricator.wikimedia.org/T435554) [10:47:16] (03CR) 10Laurabarluzzi: Add missing valid ISO countries to Gravy to stop PayPal falling back to paypal_ec (031 comment) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1331812 (https://phabricator.wikimedia.org/T435554) (owner: 10Laurabarluzzi) [13:17:11] FIRING: NodeDown: Node payments1012 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=payments1012 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [13:22:11] RESOLVED: NodeDown: Node payments1012 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=payments1012 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [14:29:42] 14Fundraising Sprint: Kondo-approved bare room, 06Fundraising-Backlog: Sprint L Prospects - https://phabricator.wikimedia.org/T430364#12271259 (10greg) 05Open→03Resolved a:03greg [14:30:30] 14Fundraising Sprint: Kondo-approved bare room, 06Fundraising-Backlog: Sprint L Prospects - https://phabricator.wikimedia.org/T430364#12271267 (10greg) a:05greg→03None [14:31:41] (03PS5) 10Laurabarluzzi: Add missing valid ISO countries to Gravy to stop PayPal falling back to paypal_ec [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1331812 (https://phabricator.wikimedia.org/T435554) [14:40:49] (03CR) 10Laurabarluzzi: Add missing valid ISO countries to Gravy to stop PayPal falling back to paypal_ec (031 comment) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1331812 (https://phabricator.wikimedia.org/T435554) (owner: 10Laurabarluzzi) [14:43:51] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Consider bounce handling for emails from CiviCRM - https://phabricator.wikimedia.org/T436528 (10Lars) 03NEW [15:24:31] (03PS1) 10Ejegg: Fix payment_attempt logging exclusion [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332758 [15:24:31] (03PS1) 10Ejegg: Fix payment_attempt logging exclusion [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332759 [15:29:21] (03PS1) 10Ejegg: Donation Modify - handle missing contributions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332760 (https://phabricator.wikimedia.org/T434656) [15:32:15] (03PS2) 10Ejegg: Fix payment_attempt logging exclusion [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332759 [15:32:36] (03Abandoned) 10Ejegg: Fix payment_attempt logging exclusion [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332758 (owner: 10Ejegg) [15:33:09] ah dang, that got an unrelated change squashed in [15:34:10] (03PS2) 10Ejegg: Donation Modify - handle missing contributions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332760 (https://phabricator.wikimedia.org/T434656) [15:34:30] (03PS3) 10Ejegg: Fix payment_attempt logging exclusion [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332759 [15:35:11] ok, PS3 should have it right if you want to review, larssandergreen [15:38:21] ejegg: probably not a priority to deploy this trigger update, so should we maybe just leave off the drop table and add it to a phab to do in maintenance week [15:41:01] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Consider bounce handling for emails from CiviCRM - https://phabricator.wikimedia.org/T436528#12271623 (10Ejegg) We do have bounce handling for TY mail, at least. An example from today: https://civicrm.wikimedia.org/civicrm/activity?atype=170&action=view&r... [15:43:35] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Consider bounce handling for emails from CiviCRM - https://phabricator.wikimedia.org/T436528#12271676 (10Ejegg) Here's the custom code to add CiviMail queue items and VERP headers: https://phabricator.wikimedia.org/diffusion/WFCG/browse/master/ext/wmf-tha... [15:44:32] larssandergreen: ok, I can split the drop table into its own patch [15:44:56] actually, since there are no changes to the other triggers we could just drop those three without stopping any queues or anything [15:46:09] I think you had another triggers update before this though, can't remember if that one needed downtime [15:47:46] oh, API4 has mailing event queue entities [15:48:01] I can get rid of all that CiviMailStore custom code [15:48:15] (I think) [15:51:02] ejegg: that other one was just an add as well, so I think we are ok to do it now if you want [15:51:11] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-AutoTY-Email, 07Technical-Debt: Remove CiviMailStore, use API4 - https://phabricator.wikimedia.org/T436537 (10Ejegg) 03NEW [15:51:46] Jeff_Green: we just realized we are logging a table that we meant to exclude. [15:52:03] civicrm_payment_attempts has a fair amount of data we don't need to duplicate [15:52:11] so we'd like to drop a few triggers [15:52:29] it shouldn't need any downtime or have any effect on other tables [15:52:51] (03CR) 10Lars SG: [C:03+2] Fix payment_attempt logging exclusion [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332759 (owner: 10Ejegg) [15:53:29] just these statements: [15:53:30] DROP TRIGGER IF EXISTS civicrm_payment_attempt_before_insert // [15:53:30] DROP TRIGGER IF EXISTS civicrm_payment_attempt_before_update // [15:53:31] DROP TRIGGER IF EXISTS civicrm_payment_attempt_before_delete // [15:53:42] err, with the traditional delimiter I guess [15:54:35] then we can deploy that patch Lars just C+2ed to drop the log_civicrm_payment_attempt table [15:55:31] the patch also updates the triggers.mysql file to remove those triggers [15:55:41] but there's no need to re-run the whole file [15:58:41] ejegg: ok, perhaps we'll wait until off-hours just to be super safe since it's nonurgent [16:01:42] ok, sure thing [16:11:38] (03Merged) 10jenkins-bot: Fix payment_attempt logging exclusion [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332759 (owner: 10Ejegg) [16:18:35] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap: Adding new anonymous flag to Chariot email alerts - https://phabricator.wikimedia.org/T433441#12271849 (10MDemosWMF) @Eileenmcnaughton The field name in Chariot that will have the Endowment and Anonymous flags has been updated to "Flag" - the policy in Chariot... [16:19:28] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Consider bounce handling for emails from CiviCRM - https://phabricator.wikimedia.org/T436528#12271864 (10Lars) Thanks @Ejegg, I guess this task is then to generalize that to all transactional emails (and also to enable it for CiviMail). [16:20:25] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Consider bounce handling for emails from CiviCRM - https://phabricator.wikimedia.org/T436528#12271869 (10Lars) And to make those bounces visible in a report. [16:22:05] ejegg: this one might be a good one for you to review: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1332533 [16:45:51] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Change highest donation fields in Acoustic export to use native currency not USD - https://phabricator.wikimedia.org/T436542 (10Lars) 03NEW [16:47:53] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Change highest donation fields in Acoustic export to use native currency not USD - https://phabricator.wikimedia.org/T436542#12271981 (10Lars) Need to consider how we handle the case where the donor has given in more than once currency. [16:53:42] oh nice larssandergreen I was hoping someone was working on deadlock requeueing! [16:59:20] arg that $ageLimit is a lie [16:59:30] when queues have been off for a while, anyway [17:00:31] old old phab, we should probably actually do something about it: https://phabricator.wikimedia.org/T192692 [17:01:02] cripes, WIP fix filed in 2018 and abandoned in 2024 [17:01:46] will finish reviewing after some lunch [17:12:14] 03Fundraising Sprint: Pedestal to the Metal, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Change AF_recurring_latest_currency & AF_recurring_latest_native_amount to reflect latest donation - https://phabricator.wikimedia.org/T436188#12272044 (10Lars) SQL update before deployment: ` ALTER TABLE s... [17:12:52] 03Fundraising Sprint: Pedestal to the Metal, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Change AF_recurring_latest_currency & AF_recurring_latest_native_amount to reflect latest donation - https://phabricator.wikimedia.org/T436188#12272046 (10Lars) [17:12:53] 03Fundraising Sprint: Pedestal to the Metal, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add overall_latest fields to Acoustic export - https://phabricator.wikimedia.org/T436147#12272048 (10Lars) →14Duplicate dup:03T436188 [17:13:25] 03Fundraising Sprint: Pedestal to the Metal, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Change AF_recurring_latest_currency & AF_recurring_latest_native_amount to reflect latest donation & add overall fields - https://phabricator.wikimedia.org/T436188#12272055 (10Lars) [17:45:55] ejegg: yes, I noticed that too and decided it was out of scope, but forgot to make a phab - turns out I didn't need to since you have one from way back when [17:46:11] FIRING: NodeDown: Node frdev1003 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdev1003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [17:49:59] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-failmail: Sudden invalid_payment_method - https://phabricator.wikimedia.org/T436503#12272186 (10Laurabarluzzi) Update from Gravy: > We checked the logs on our end for the failed transactions. In every failed attempt (both network token and PAN), t... [17:51:11] RESOLVED: NodeDown: Node frdev1003 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdev1003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [17:56:43] (03PS1) 10Lars SG: Change AF_recurring_latest_currency & AF_recurring_latest_native_amount to reflect latest donation & add overall_latest fields [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1332775 [18:06:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [18:12:04] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Update MW export to send relationship type directly - https://phabricator.wikimedia.org/T436548 (10Lars) 03NEW [18:12:25] 03Fundraising Sprint: Pedestal to the Metal, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Update MW export to send relationship type directly - https://phabricator.wikimedia.org/T436548#12272269 (10Lars) [18:15:27] 03Fundraising Sprint: Pedestal to the Metal, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 05FY26-27 WE5.4 Donor Identification and recognition: Update MW export to send relationship type directly - https://phabricator.wikimedia.org/T436548#12272276 (10Jdlrobson-WMF) [18:20:39] 03Fundraising Sprint: Pedestal to the Metal, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 05FY26-27 WE5.4 Donor Identification and recognition: Update MW export to send relationship type directly - https://phabricator.wikimedia.org/T436548#12272298 (10Jdlrobson-WMF) p:05Triage→03High [18:24:07] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-failmail, 13Patch-For-Review: [TypeError] Civi\WMFQueueMessage\DonationModifyMessage::getContribution(): Return value must be of type array, null returned - https://phabricator.wikimedia.org/T434656#12272302 (10Laurabarluzzi) a:03Ejegg [18:26:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [18:36:14] (03CR) 10Ejegg: "This looks great! I would just like one more test that hits the halt condition, to check that the halted message does indeed still remain " [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332533 (https://phabricator.wikimedia.org/T436157) (owner: 10Lars SG) [18:38:02] (03PS3) 10Ejegg: Donation Modify - handle missing contributions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332760 (https://phabricator.wikimedia.org/T434656) [18:38:02] (03PS1) 10Ejegg: Donation Modify - don't double-record failures [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332781 (https://phabricator.wikimedia.org/T434656) [18:40:20] (03PS2) 10Ejegg: Donation Modify - don't double-record failures [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332781 (https://phabricator.wikimedia.org/T434656) [18:45:11] (03PS1) 10Wfan: Add currency Symbol helper for combo wiki style [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1332783 (https://phabricator.wikimedia.org/T432974) [18:53:07] (03PS1) 10Lars SG: Update MW export to send relationship type directly [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1332785 [18:53:28] 03Fundraising Sprint: Pedestal to the Metal, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 05FY26-27 WE5.4 Donor Identification and recognition: Update MW export to send relationship type directly - https://phabricator.wikimedia.org/T436548#12272428 (10Lars) a:03Lars [18:56:03] (03PS2) 10Lars SG: Update MW export to send relationship type directly [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1332785 (https://phabricator.wikimedia.org/T436548) [19:10:37] (03PS3) 10Wfan: Update combo wiki currency to use country [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1331694 (https://phabricator.wikimedia.org/T432974) [19:12:20] (03CR) 10CI reject: [V:04-1] Update combo wiki currency to use country [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1331694 (https://phabricator.wikimedia.org/T432974) (owner: 10Wfan) [19:17:12] (03PS4) 10Wfan: Update combo wiki currency to use country [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1331694 (https://phabricator.wikimedia.org/T432974) [19:17:41] (03PS5) 10Wfan: Update combo wiki currency to use country [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1331694 (https://phabricator.wikimedia.org/T432974) [19:19:09] (03CR) 10CI reject: [V:04-1] Update combo wiki currency to use country [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1331694 (https://phabricator.wikimedia.org/T432974) (owner: 10Wfan) [19:19:56] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-failmail, 13Patch-For-Review: [TypeError] Civi\WMFQueueMessage\DonationModifyMessage::getContribution(): Return value must be of type array, null returned - https://phabricator.wikimedia.org/T434656#12272471 (10Laurabarluzzi) @Ejegg writing here... [19:19:58] (03CR) 10Laurabarluzzi: [C:03+2] "This handles the situation of the contribution not being found, and it looks good. I dropped a comment in the phab regarding the status of" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332760 (https://phabricator.wikimedia.org/T434656) (owner: 10Ejegg) [19:24:56] (03CR) 10Laurabarluzzi: Donation Modify - don't double-record failures (032 comments) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332781 (https://phabricator.wikimedia.org/T434656) (owner: 10Ejegg) [19:35:29] (03PS6) 10Wfan: Update combo wiki currency to use country [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1331694 (https://phabricator.wikimedia.org/T432974) [19:36:18] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint: Merging contacts can create multiple segment/status change rows in wmf_donor_history - https://phabricator.wikimedia.org/T431741#12272533 (10XenoRyet) [19:37:23] (03CR) 10CI reject: [V:04-1] Update combo wiki currency to use country [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1331694 (https://phabricator.wikimedia.org/T432974) (owner: 10Wfan) [19:37:27] 03Fundraising Sprint: Pedestal to the Metal, 07fr-combowiki: fix the ty page params from combo wiki - https://phabricator.wikimedia.org/T436165#12272544 (10AnnWF) a:03AnnWF [19:37:43] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-failmail: Sudden invalid_payment_method - https://phabricator.wikimedia.org/T436503#12272548 (10XenoRyet) 05Open→03In progress p:05Triage→03Low [19:38:08] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: FormBuilder pre and post help for a field aren't editable - https://phabricator.wikimedia.org/T436279#12272552 (10XenoRyet) p:05Triage→03Low [19:38:50] (03Merged) 10jenkins-bot: Donation Modify - handle missing contributions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332760 (https://phabricator.wikimedia.org/T434656) (owner: 10Ejegg) [19:39:30] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Double transactions are happening with the same exact order id, first one failing but second succeeds - https://phabricator.wikimedia.org/T436046#12272568 (10XenoRyet) p:05Triage→03High [19:41:23] 06Fundraising-Backlog: Get our ID in the checkout.com console - https://phabricator.wikimedia.org/T431628#12272569 (10Cstone) Gravy just added this for us they are passing through our id in the udf4 column We need to test this [19:43:41] 03Fundraising Sprint: Pedestal to the Metal, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Change highest donation fields in Acoustic export to use native currency not USD - https://phabricator.wikimedia.org/T436542#12272575 (10Lars) [19:43:57] 03Fundraising Sprint: Pedestal to the Metal, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Change highest donation fields in Acoustic export to use native currency not USD - https://phabricator.wikimedia.org/T436542#12272577 (10Lars) a:03Lars [19:47:21] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-WMF-Audit: Trustly not clearing because backend_processor_trxn_id has gravy ID in it not trustly... - https://phabricator.wikimedia.org/T436050#12272607 (10AKanji-WMF) p:05Triage→03Medium [19:49:01] (03PS7) 10Wfan: Update combo wiki currency to use country [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1331694 (https://phabricator.wikimedia.org/T432974) [19:49:07] 06Fundraising-Backlog, 10FR-WMF-Audit: Adyen audit failed because transaction set to Cancelled but came through as a settled chargeback - https://phabricator.wikimedia.org/T436318#12272619 (10Eileenmcnaughton) Just updated another one to allow the file to clear https://civicrm.wikimedia.org/civicrm/contact/vie... [19:51:04] (03CR) 10CI reject: [V:04-1] Update combo wiki currency to use country [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1331694 (https://phabricator.wikimedia.org/T432974) (owner: 10Wfan) [19:51:10] 06Fundraising-Backlog, 10FR-WMF-Audit: Adyen audit failed because transaction set to Cancelled but came through as a settled chargeback - https://phabricator.wikimedia.org/T436318#12272627 (10Eileenmcnaughton) and 148528682 [19:51:11] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 10Recurring-Donations: Trustly subsequent recurring fails with invalid_service_credentials - https://phabricator.wikimedia.org/T434933#12272628 (10Cstone) a:05Cstone→03None @Damilare I don't know what I did when I assigned the... [19:51:58] (03PS8) 10Wfan: Update combo wiki currency to use country [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1331694 (https://phabricator.wikimedia.org/T432974) [19:53:44] (03CR) 10CI reject: [V:04-1] Update combo wiki currency to use country [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1331694 (https://phabricator.wikimedia.org/T432974) (owner: 10Wfan) [19:59:31] 06Fundraising-Backlog: Evaluate how to best handle changes to payment methods for recurring contributions in Gravy - https://phabricator.wikimedia.org/T436299#12272664 (10Cstone) We should: [ ] Look into how to just push these right to the first failure email, can we distinguish these from others [ ] Ask gravy m... [20:01:51] 06Fundraising-Backlog: Evaluate how to best handle changes to payment methods for recurring contributions in Gravy - https://phabricator.wikimedia.org/T436299#12272682 (10AKanji-WMF) @RKumar_WMF can you please add this to the gravy issue tracker? [20:13:07] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-WMF-Audit: Adyen audit failed because transaction set to Cancelled but came through as a settled chargeback - https://phabricator.wikimedia.org/T436318#12272703 (10AKanji-WMF) p:05Triage→03High FYI this patch we committed changed how we listen... [20:15:30] (03CR) 10Eileen: [C:03+2] Add activity when sending new checksum link [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1329650 (https://phabricator.wikimedia.org/T436160) (owner: 10Lars SG) [20:17:24] (03CR) 10Eileen: [C:03+2] Clear cancel date, cancel reason and end date when re-activating a Paypal recurring [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1329671 (https://phabricator.wikimedia.org/T436172) (owner: 10Lars SG) [20:18:15] (03CR) 10Eileen: [C:03+2] Set a default silverpop database id instead of mocking it in each test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1331963 (https://phabricator.wikimedia.org/T433866) (owner: 10Lars SG) [20:19:40] (03PS1) 10Lars SG: Change highest donation fields to use native currency not USD [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1332805 [20:19:56] (03CR) 10Eileen: [C:03+2] REF: Move createContribution from ThankYouTest to WMFEnvironmentTrait for general use [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1331867 (https://phabricator.wikimedia.org/T433866) (owner: 10Lars SG) [20:20:08] 03Fundraising Sprint: Pedestal to the Metal, 06Fundraising-Backlog, 07payments-orchestration: Get our ID in the checkout.com console - https://phabricator.wikimedia.org/T431628#12272730 (10AKanji-WMF) [20:23:22] 03Fundraising Sprint: Pedestal to the Metal, 06Fundraising-Backlog, 07payments-orchestration: Get our ID in the checkout.com console - https://phabricator.wikimedia.org/T431628#12272742 (10Cstone) a:03Cstone [20:23:37] thanks for the review eileen. I'm hoping to get these two reviewed and deployed today if possible, email folks are eager for these fixes for email later this week: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/tools/+/1332805 [20:24:00] and https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/tools/+/1332775 and (less important, but quick review) https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/tools/+/1332785 [20:28:41] 06Fundraising-Backlog, 07fr-combowiki: fix the ty page params from combo wiki - https://phabricator.wikimedia.org/T436165#12272758 (10AnnWF) [20:33:45] (03Merged) 10jenkins-bot: Add activity when sending new checksum link [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1329650 (https://phabricator.wikimedia.org/T436160) (owner: 10Lars SG) [20:37:15] (03Merged) 10jenkins-bot: Clear cancel date, cancel reason and end date when re-activating a Paypal recurring [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1329671 (https://phabricator.wikimedia.org/T436172) (owner: 10Lars SG) [20:41:12] (03Merged) 10jenkins-bot: REF: Move createContribution from ThankYouTest to WMFEnvironmentTrait for general use [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1331867 (https://phabricator.wikimedia.org/T433866) (owner: 10Lars SG) [20:41:15] (03Merged) 10jenkins-bot: Set a default silverpop database id instead of mocking it in each test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1331963 (https://phabricator.wikimedia.org/T433866) (owner: 10Lars SG) [20:50:11] FIRING: NodeDown: Node frav1003 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frav1003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:55:11] RESOLVED: NodeDown: Node frav1003 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frav1003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [21:21:10] (03PS2) 10Lars SG: Improve deadlock handling in QueueConsumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332533 (https://phabricator.wikimedia.org/T436157) [21:22:35] (03CR) 10Lars SG: "I expanded the tests to make sure we're covering in detail the expected re-queue / rollback behaviour on halting." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332533 (https://phabricator.wikimedia.org/T436157) (owner: 10Lars SG) [21:28:49] pulling down the latest ^^ [21:33:26] (03PS3) 10Lars SG: Improve deadlock handling in QueueConsumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332533 (https://phabricator.wikimedia.org/T436157) [21:34:05] (03CR) 10Ejegg: [C:03+2] "Thanks, this looks great! Here's hoping it makes a dent in the chaos crew workload." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332533 (https://phabricator.wikimedia.org/T436157) (owner: 10Lars SG) [21:52:54] (03Merged) 10jenkins-bot: Improve deadlock handling in QueueConsumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1332533 (https://phabricator.wikimedia.org/T436157) (owner: 10Lars SG) [22:00:19] 03Fundraising Sprint: Pedestal to the Metal, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Change AF_recurring_latest_currency & AF_recurring_latest_native_amount to reflect latest donation & add overall fields - https://phabricator.wikimedia.org/T436188#12273090 (10Dwisehaupt) Query complete: ` Ma... [23:14:29] 06Fundraising-Backlog, 10fundraising-tech-ops: Staging db refresh - https://phabricator.wikimedia.org/T436146#12273273 (10Dwisehaupt) 05Open→03Resolved Restore completed at 20260830-235342 ` LOAD: fredge.20260824-133407.frdb2004.sql.gz 20260826-171233 FINISHED: 20260826-213244 LOAD: civicrm.20260824-09... [23:18:47] 06Fundraising-Backlog, 10fundraising-tech-ops: Staging db refresh - https://phabricator.wikimedia.org/T436146#12273278 (10Dwisehaupt) Also hand restored smashpig to dev_smashpig since that was requested. `zcat smashpig.20260824-134632.frdb2004.sql.gz | mariadb dev_smashpig` [23:55:25] (03PS5) 10Dmiranda: Identify moto transactions and keep their metadata [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1327676 (https://phabricator.wikimedia.org/T432672)