[00:11:38] (03PS1) 10Eileen: Upgrade smashPig, gravy [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1169270 (https://phabricator.wikimedia.org/T394334) [00:12:50] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MW-1.45-notes (1.45.0-wmf.10; 2025-07-15), 13Patch-For-Review: Remove Gravy fiscal_number countries requirement from PayPal transactions - https://phabricator.wikimedia.org/T394334#11002812 (10Eileenmcnaughton) We just had another one of these https... [00:37:59] (03PS1) 10Eileen: Remove data axle import job [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1169271 [02:09:54] (03PS1) 10Eileen: Fix loading of contact mappings [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1169276 (https://phabricator.wikimedia.org/T392252) [02:10:33] 03Fundraising Sprint Michelin Star, 06Fundraising-Backlog, 13Patch-For-Review: Direct Mail NCOA Updates - https://phabricator.wikimedia.org/T392252#11002997 (10Eileenmcnaughton) [05:35:02] 06Fundraising-Backlog: Failed, Processing, Pending transactions are showing in Civi as completed. - https://phabricator.wikimedia.org/T399527 (10RKumar_WMF) 03NEW [06:52:11] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_43) - 10https://gerrit.wikimedia.org/r/1169399 (owner: 10L10n-bot) [08:19:58] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences: Comms preference centre - scope adding to Acoustic email footers - https://phabricator.wikimedia.org/T392742#11003350 (10ppenloglou) Ah thanks for breaking down the checksum! I just tried again in Ac... [10:32:09] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MW-1.45-notes (1.45.0-wmf.10; 2025-07-15), 13Patch-For-Review: Remove Gravy fiscal_number countries requirement from PayPal transactions - https://phabricator.wikimedia.org/T394334#11003840 (10RKumar_WMF) Since we are on direct connection with Paypa... [11:09:19] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_39) - 10https://gerrit.wikimedia.org/r/1168739 (owner: 10L10n-bot) [11:10:12] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_43) - 10https://gerrit.wikimedia.org/r/1169399 (owner: 10L10n-bot) [12:35:52] (03PS1) 10Jgleeson: Refactor: Clean up array formatting in RecordCaptureJob [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169646 [13:16:54] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Bank Transfer Transactions: Status Mismatch (Gr4vy/dLocal vs. CiviCRM) - https://phabricator.wikimedia.org/T399577 (10jgleeson) 03NEW [13:18:41] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Bank Transfer Transactions: Status Mismatch (Gr4vy/dLocal vs. CiviCRM) - https://phabricator.wikimedia.org/T399577#11004392 (10jgleeson) [13:19:14] argh. chaos is strong today [13:20:47] (03PS1) 10Jgleeson: Refactor: Improve RecordCaptureJob log message clarity [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169652 (https://phabricator.wikimedia.org/T399577) [13:21:49] (03PS2) 10Jgleeson: Refactor: Improve RecordCaptureJob log message clarity [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169652 (https://phabricator.wikimedia.org/T399577) [13:31:20] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Bank Transfer Transactions: Status Mismatch (Gr4vy/dLocal vs. CiviCRM) - https://phabricator.wikimedia.org/T399577#11004448 (10jgleeson) [14:12:22] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences: Comms pref centre shows "log in" - https://phabricator.wikimedia.org/T399581 (10AKanji-WMF) 03NEW [14:43:25] (03PS3) 10Thiemo Kreuz (WMDE): build: Updating mediawiki/mediawiki-phan-config to 0.16.0 [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1168019 (owner: 10Libraryupgrader) [14:46:08] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences: Comms preference centre - scope adding to Acoustic email footers - https://phabricator.wikimedia.org/T392742#11004759 (10AKanji-WMF) Thank you both - noting I am trying to add relevant comments to [[... [16:13:21] 10fundraising-tech-ops: Superset and Metabase access for JTanner - https://phabricator.wikimedia.org/T399490#11005241 (10AStein-WMF) You should have received an email with your new ssl cert and a text with the password! please let me know if you need help installing it! [16:16:01] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Adyen daily audit file check - https://phabricator.wikimedia.org/T399034#11005287 (10MBeat33) [16:29:39] 10BDC-Implementation: implement dagster logging - https://phabricator.wikimedia.org/T392388#11005439 (10AStein-WMF) dagster is logging to syslog on the analytics host [16:29:41] 10BDC-Implementation: implement dagster logging - https://phabricator.wikimedia.org/T392388#11005440 (10AStein-WMF) 05Open→03Resolved [16:31:43] (03PS1) 10Jgleeson: Add temporary debug logging for T399577 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169698 (https://phabricator.wikimedia.org/T399577) [16:34:18] fr-tech that should allow us to temporarily inspect the gravy webhooks sent over. I'll try it local first after dinner and then push it out if no one is around to review. [16:46:29] 10fundraising-tech-ops: Fundraising access request for Hawa Ahmed - https://phabricator.wikimedia.org/T399009#11005568 (10AStein-WMF) [16:47:23] 10fundraising-tech-ops: Fundraising access request for Hawa Ahmed - https://phabricator.wikimedia.org/T399009#11005572 (10AStein-WMF) We confirmed that Hawa can access superset and has working ssh/ yubikey access! Marking this as complete [16:47:41] 10fundraising-tech-ops: Fundraising access request for Hawa Ahmed - https://phabricator.wikimedia.org/T399009#11005573 (10AStein-WMF) 05Open→03Resolved [18:07:52] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM: Meeting type fields on MG activities - https://phabricator.wikimedia.org/T381535#11005968 (10DBautista-WMF) @AKanji-WMF no problem, confirming that this request can wait. Thank you! [18:09:26] (03PS2) 10Jgleeson: Add temporary debug logging for T399577 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169698 (https://phabricator.wikimedia.org/T399577) [18:17:24] (03CR) 10Jgleeson: [C:03+2] "self-merging due to pushing out temp debugging and no one else around atm" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169646 (owner: 10Jgleeson) [18:17:31] (03CR) 10Jgleeson: [C:03+2] "self-merging due to pushing out temp debugging and no one else around atm" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169652 (https://phabricator.wikimedia.org/T399577) (owner: 10Jgleeson) [18:17:40] (03CR) 10Jgleeson: [C:03+2] "self-merging due to pushing out temp debugging and no one else around atm" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169698 (https://phabricator.wikimedia.org/T399577) (owner: 10Jgleeson) [18:18:06] (03Merged) 10jenkins-bot: Refactor: Clean up array formatting in RecordCaptureJob [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169646 (owner: 10Jgleeson) [18:18:48] (03Merged) 10jenkins-bot: Refactor: Improve RecordCaptureJob log message clarity [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169652 (https://phabricator.wikimedia.org/T399577) (owner: 10Jgleeson) [18:18:52] (03Merged) 10jenkins-bot: Add temporary debug logging for T399577 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169698 (https://phabricator.wikimedia.org/T399577) (owner: 10Jgleeson) [18:58:38] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Gravy Card Payment AVS check triggering false positives. - https://phabricator.wikimedia.org/T396355#11006166 (10jgleeson) Hi @EMartin. Thanks for helping with the testing. Visa transactions appear to use different AVS codes than the ones related to this... [19:13:29] (03PS1) 10Jgleeson: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1169738 [19:16:11] (03CR) 10Jgleeson: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1169738 (owner: 10Jgleeson) [19:16:38] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1169738 (owner: 10Jgleeson) [19:17:51] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences: Comms preference centre - scope adding to Acoustic email footers - https://phabricator.wikimedia.org/T392742#11006201 (10AKanji-WMF) Thanks @ppenloglou - good to go to incorporate into Acoustic email... [19:25:25] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Comms pref centre shows "log in" - https://phabricator.wikimedia.org/T399581#11006226 (10AKanji-WMF) p:05Triage→03Medium Likely requires change to display config. Putting in chaos for review. [19:28:27] (03PS1) 10Jgleeson: Use print_r() for better debug object visibility [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169743 (https://phabricator.wikimedia.org/T399577) [19:32:58] 06Fundraising-Backlog, 07payments-orchestration: Discuss how we will handle Gravy's descriptor limit - https://phabricator.wikimedia.org/T398652#11006245 (10AKanji-WMF) From the referenced phab and discussion "Today we send "Wikimedia 877 600 9454" as a dynamic value that's being passed in the payment request.... [19:34:11] 03Fundraising Sprint Michelin Star: Documentation: common pref center explination - https://phabricator.wikimedia.org/T399622 (10AnnWF) 03NEW [19:37:08] (03CR) 10Wfan: "recheck" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1165576 (owner: 10Wfan) [19:40:33] (03CR) 10Jgleeson: [C:03+2] Upgrade smashPig, gravy [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1169270 (https://phabricator.wikimedia.org/T394334) (owner: 10Eileen) [19:40:40] (03PS2) 10Wfan: Remove epc checksum valid check when consume [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1167965 (https://phabricator.wikimedia.org/T398381) [19:40:51] (03CR) 10Jgleeson: [C:03+2] Use print_r() for better debug object visibility [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169743 (https://phabricator.wikimedia.org/T399577) (owner: 10Jgleeson) [19:46:37] (03CR) 10Jgleeson: [C:03+2] Fix loading of contact mappings [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1169276 (https://phabricator.wikimedia.org/T392252) (owner: 10Eileen) [19:54:51] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Adyen daily audit file check - https://phabricator.wikimedia.org/T399034#11006326 (10EMartin) Do we know why it was missed as we'd like to use this manual facility for large gifts again in the future? [19:57:00] (03Merged) 10jenkins-bot: Upgrade smashPig, gravy [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1169270 (https://phabricator.wikimedia.org/T394334) (owner: 10Eileen) [19:57:02] (03Merged) 10jenkins-bot: Use print_r() for better debug object visibility [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169743 (https://phabricator.wikimedia.org/T399577) (owner: 10Jgleeson) [19:58:11] (03PS1) 10Jgleeson: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1169757 [19:58:32] (03CR) 10Jgleeson: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1169757 (owner: 10Jgleeson) [20:02:25] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog: Sprint N - Priorities [2025] - https://phabricator.wikimedia.org/T399121#11006367 (10AKanji-WMF) [20:02:32] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint: Add browserInfo parameters to Gravy for Adyen 3DS flow compatibility - https://phabricator.wikimedia.org/T398845#11006368 (10AKanji-WMF) [20:02:35] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint: Map consolidated Gr4vy file to Civi field and table locations - https://phabricator.wikimedia.org/T397662#11006371 (10AKanji-WMF) [20:02:36] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint: Benevity - moving possible duplicate errors to group for dedupe - https://phabricator.wikimedia.org/T396535#11006372 (10AKanji-WMF) [20:02:40] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07payments-orchestration: Discuss how we will handle Gravy's descriptor limit - https://phabricator.wikimedia.org/T398652#11006369 (10AKanji-WMF) [20:02:44] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07payments-orchestration: Start updating the fee information into Civi once we get it - https://phabricator.wikimedia.org/T396071#11006373 (10AKanji-WMF) [20:02:48] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-acoustic: July 30th: Removing Data Axle CE fields from Civi - License Expiring - https://phabricator.wikimedia.org/T395961#11006374 (10AKanji-WMF) [20:02:50] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint, 13Patch-For-Review: Implement Gravy payment method deletion handling for subscription cancellations - https://phabricator.wikimedia.org/T398723#11006370 (10AKanji-WMF) [20:02:50] (03Merged) 10jenkins-bot: Fix loading of contact mappings [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1169276 (https://phabricator.wikimedia.org/T392252) (owner: 10Eileen) [20:02:53] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint: Civi recurring donations display question - https://phabricator.wikimedia.org/T393227#11006375 (10AKanji-WMF) [20:02:57] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1169757 (owner: 10Jgleeson) [20:03:01] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint, 07payments-orchestration: Enable SEPA via Gr4vy for EU countries. - https://phabricator.wikimedia.org/T392065#11006378 (10AKanji-WMF) [20:03:05] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint: Payment method migration: Credit cards - Import tokens to Civi - https://phabricator.wikimedia.org/T391991#11006379 (10AKanji-WMF) [20:03:09] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint: Create donor portal page for recurring donation pause/cancel process - https://phabricator.wikimedia.org/T388749#11006381 (10AKanji-WMF) [20:03:13] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 10FR-Donor-portal, 10FR-email-preferences, 07fr-current-sprint: Create donor portal main page displaying info from CiviCRM - https://phabricator.wikimedia.org/T388745#11006382 (10AKanji-WMF) [20:03:17] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint: Setting up consolidated settlement reporting for all PSPs via Gr4vy for new recon process - https://phabricator.wikimedia.org/T391496#11006380 (10AKanji-WMF) [20:03:21] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint: Comms preference centre - email duplication check - https://phabricator.wikimedia.org/T385911#11006385 (10AKanji-WMF) [20:03:25] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 13Patch-For-Review: Direct Mail NCOA Updates - https://phabricator.wikimedia.org/T392252#11006377 (10AKanji-WMF) [20:03:29] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint: Allow donors to request a donor portal login link by email - https://phabricator.wikimedia.org/T388739#11006383 (10AKanji-WMF) [20:03:33] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint: Gravy Dlocal - https://phabricator.wikimedia.org/T384967#11006397 (10AKanji-WMF) [20:03:37] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint: Fix dedupe merge wrapper script to track modified date - https://phabricator.wikimedia.org/T383194#11006396 (10AKanji-WMF) [20:03:41] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint, 13Patch-For-Review: How will we expose payment processor errors/issues with gravy - https://phabricator.wikimedia.org/T387411#11006384 (10AKanji-WMF) [20:03:45] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 06FR-donorrelations: PIX and Boleto after completing the donation do not show TY page - https://phabricator.wikimedia.org/T394979#11006399 (10AKanji-WMF) [20:03:53] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 07fr-current-sprint: Enable iDeal via Gr4vy in Netherlands - https://phabricator.wikimedia.org/T392068#11006402 (10AKanji-WMF) [20:03:57] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-acoustic, 07fr-current-sprint, 13Patch-For-Review: Develop process for tracking & handling SMS consent records - https://phabricator.wikimedia.org/T379702#11006400 (10AKanji-WMF) [20:04:01] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 06FR-donorrelations, 07fr-current-sprint: Gravy & the PayPal recurring cancellation process - https://phabricator.wikimedia.org/T393226#11006401 (10AKanji-WMF) [20:04:09] 10Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint, 13Patch-For-Review: Standalone dev builds - enable display_errors - https://phabricator.wikimedia.org/T396171#11006403 (10AKanji-WMF) [20:07:05] !log SmashPig upgraded from 82dbf9ff to 0bc0c1ec [20:07:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:07:25] 14Fundraising Sprint Michelin Star, 06Fundraising-Backlog: 2nd Recurring Failure Email in Civi - https://phabricator.wikimedia.org/T365488#11006422 (10AKanji-WMF) [20:07:26] 06Fundraising-Backlog, 07fr-acoustic: Export phone numbers to Acoustic - https://phabricator.wikimedia.org/T380946#11006421 (10AKanji-WMF) [20:07:28] 06Fundraising-Backlog: Duplicates after batch data entry - https://phabricator.wikimedia.org/T382516#11006423 (10AKanji-WMF) [20:07:29] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations, 07Unplanned-Sprint-Work: Civi merge is not saving the state when a mailing address is present. - https://phabricator.wikimedia.org/T379684#11006424 (10AKanji-WMF) [20:08:59] 03Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint: Sprint N - Priorities [2025] - https://phabricator.wikimedia.org/T399121#11006435 (10AKanji-WMF) [20:09:01] 03Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint, 07payments-orchestration: Discuss how we will handle Gravy's descriptor limit - https://phabricator.wikimedia.org/T398652#11006436 (10AKanji-WMF) [20:09:02] 03Fundraising Sprint: N 2025, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint, 07payments-orchestration: Start updating the fee information into Civi once we get it - https://phabricator.wikimedia.org/T396071#11006437 (10AKanji-WMF) [20:09:04] 03Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-acoustic, 07fr-current-sprint: July 30th: Removing Data Axle CE fields from Civi - License Expiring - https://phabricator.wikimedia.org/T395961#11006438 (10AKanji-WMF) [20:09:05] 03Fundraising Sprint: N 2025, 06Fundraising-Backlog, 06FR-donorrelations, 07fr-current-sprint: PIX and Boleto after completing the donation do not show TY page - https://phabricator.wikimedia.org/T394979#11006440 (10AKanji-WMF) [20:09:09] 03Fundraising Sprint: N 2025, 06Fundraising-Backlog, 07fr-current-sprint, 13Patch-For-Review: Direct Mail NCOA Updates - https://phabricator.wikimedia.org/T392252#11006439 (10AKanji-WMF) [20:11:33] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap: Add VIPPS in Norway - https://phabricator.wikimedia.org/T390870#11006457 (10AKanji-WMF) [20:23:14] 14Fundraising Sprint: UnusedFunctionButDontDeleteItIncaseItsUsed(), 06Fundraising-Backlog, 07payments-orchestration: Map Processing status for Gravy/Trustly ACH to Smashpig Complete - https://phabricator.wikimedia.org/T377376#11006535 (10jgleeson) This one came back to haunt us. A consequence of importin... [20:27:25] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Bank Transfer Transactions: Status Mismatch (Gr4vy/dLocal vs. CiviCRM) - https://phabricator.wikimedia.org/T399577#11006552 (10jgleeson) We tracked the code responsible for this bug down [[ https://github.com/wikimedia/wikimedia-fund... [20:53:42] (03PS1) 10Eileen: Update wikimedia/smash-pig gr4vy/gr4vy-php [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1169767 [20:54:22] (03CR) 10Eileen: [C:03+2] Update wikimedia/smash-pig gr4vy/gr4vy-php [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1169767 (owner: 10Eileen) [20:54:48] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1169768 [20:55:00] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1169768 (owner: 10Eileen) [21:07:25] !log civicrm upgraded from 521d0dbe to 71b58ed5 [21:07:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:07:57] !log civicrm upgraded from 521d0dbe to 71b58ed5 [21:07:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:09:04] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MW-1.45-notes (1.45.0-wmf.10; 2025-07-15), 13Patch-For-Review: Remove Gravy fiscal_number countries requirement from PayPal transactions - https://phabricator.wikimedia.org/T394334#11006696 (10Eileenmcnaughton) I just deployed Dami's patch - does an... [21:11:36] (03Merged) 10jenkins-bot: Update wikimedia/smash-pig gr4vy/gr4vy-php [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1169767 (owner: 10Eileen) [21:37:19] (03PS1) 10Jgleeson: Remove temporary debug logging for T399577 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169778 (https://phabricator.wikimedia.org/T399577) [21:37:21] (03PS1) 10Jgleeson: Fix Gravy bank payment status mapping to be payment-method-specific [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169779 (https://phabricator.wikimedia.org/T399577) [21:38:14] fr-tech, I've pushed a small patch to revert those logging debug updates, which I'll self merge and push out now before I hop off [21:38:49] I've also pushed a fix to the issue from earlier today relating to us prematurely marking Stitch payments as complete [21:40:57] (03CR) 10Jgleeson: [C:03+2] "self-merging to remove debug logging" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169778 (https://phabricator.wikimedia.org/T399577) (owner: 10Jgleeson) [21:41:50] (03PS2) 10Jgleeson: Fix Gravy bank payment status mapping to be payment-method-specific [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169779 (https://phabricator.wikimedia.org/T399577) [21:46:47] (03Merged) 10jenkins-bot: Remove temporary debug logging for T399577 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169778 (https://phabricator.wikimedia.org/T399577) (owner: 10Jgleeson) [21:48:22] (03PS3) 10Jgleeson: Remove incorrect Trustly status mapping from generic bank payment mapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169779 (https://phabricator.wikimedia.org/T399577) [21:52:40] (03PS1) 10Jgleeson: Remove temporary debug logging for T399577 [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1169780 (https://phabricator.wikimedia.org/T399577) [21:53:52] (03PS4) 10Jgleeson: Remove incorrect Trustly status mapping from generic bank payment mapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169779 (https://phabricator.wikimedia.org/T399577) [21:54:26] (03PS5) 10Jgleeson: Remove ACH-specific status mapping from bank payment mapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169779 (https://phabricator.wikimedia.org/T399577) [21:57:16] (03CR) 10Jgleeson: [C:03+2] Remove temporary debug logging for T399577 [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1169780 (https://phabricator.wikimedia.org/T399577) (owner: 10Jgleeson) [21:57:45] (03Merged) 10jenkins-bot: Remove temporary debug logging for T399577 [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1169780 (https://phabricator.wikimedia.org/T399577) (owner: 10Jgleeson) [22:01:04] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog, 07fr-current-sprint: Direct Mail NCOA Updates - https://phabricator.wikimedia.org/T392252#11006966 (10Eileenmcnaughton) a:03Eileenmcnaughton [22:01:07] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog, 07fr-current-sprint: Direct Mail NCOA Updates - https://phabricator.wikimedia.org/T392252#11006967 (10Eileenmcnaughton) [22:03:43] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog, 07fr-current-sprint: Direct Mail NCOA Updates - https://phabricator.wikimedia.org/T392252#11007007 (10MDemosWMF) Running it now and all looks good! @Eileenmcnaughton [22:04:06] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog, 07fr-current-sprint: Direct Mail NCOA Updates - https://phabricator.wikimedia.org/T392252#11007008 (10Eileenmcnaughton) 05Open→03Resolved Yay [22:07:10] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7508 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [22:08:02] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog, 07fr-current-sprint: Civi recurring donations display question - https://phabricator.wikimedia.org/T393227#11007069 (10Eileenmcnaughton) @AMJohnson I took a look and the last recurring to be created with no payment method was on 2024-08-22 05:21:57... [22:08:42] (03PS6) 10Jgleeson: Remove ACH-specific status mapping from bank payment mapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1169779 (https://phabricator.wikimedia.org/T399577) [22:12:10] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7516 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [22:17:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7516 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [22:22:10] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7519 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [22:27:10] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7526 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [22:32:10] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7528 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [22:37:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7503 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [22:42:10] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7505 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [22:47:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7506 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [22:52:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7509 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [22:57:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7511 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [23:02:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7513 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [23:07:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7518 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [23:12:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7521 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [23:17:10] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7521 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [23:22:10] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7524 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [23:27:10] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7525 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [23:32:10] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7529 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [23:37:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7506 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [23:42:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7507 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [23:43:00] mail is backing up ^ [23:43:55] https://frmon.wikimedia.org/d/sZWA7LpZk/mail?orgId=1&from=now-24h&to=now&timezone=utc&refresh=1m&viewPanel=panel-26 [23:44:23] it's leveled off but not coming down :/ [23:47:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7508 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [23:47:58] I just text Jeff [23:49:24] of course it's midnight over there [23:50:07] eileen: can you see anything weird on the civi side wrt to mail sending? [23:52:10] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7511 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [23:57:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7512 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [23:57:48] Called and left a message for Jeff [23:59:49] I don't see anything changing in the "queue delay" metric (import into civi & thankyou sent): https://frmon.wikimedia.org/goto/KnQPRKUHg?orgId=1