[01:13:08] PROBLEM - check_load on franio2002 is CRITICAL: LOAD CRITICAL - total load average: 75.76, 71.09, 36.83 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2002&service=check_load [01:18:08] PROBLEM - check_load on franio2001 is CRITICAL: LOAD CRITICAL - total load average: 18.13, 27.83, 20.85 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2001&service=check_load [01:18:08] PROBLEM - check_load on franio2002 is CRITICAL: LOAD CRITICAL - total load average: 0.90, 28.14, 27.86 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2002&service=check_load [01:23:08] PROBLEM - check_load on franio2002 is CRITICAL: LOAD CRITICAL - total load average: 0.03, 10.33, 20.17 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2002&service=check_load [01:23:08] PROBLEM - check_load on franio2001 is CRITICAL: LOAD CRITICAL - total load average: 0.77, 12.88, 16.61 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2001&service=check_load [01:28:08] PROBLEM - check_load on franio2002 is CRITICAL: LOAD CRITICAL - total load average: 4.59, 6.47, 15.72 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2002&service=check_load [01:31:22] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Better "Maximum impressions" any individual will see - https://phabricator.wikimedia.org/T421662 (10Krinkle) 03NEW [01:31:38] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 07fr-current-sprint: CentralNotice - Scoping Immediate Issues - https://phabricator.wikimedia.org/T417064#11762617 (10Krinkle) [01:33:54] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 07fr-current-sprint: CentralNotice - Scoping Immediate Issues - https://phabricator.wikimedia.org/T417064#11762618 (10Krinkle) [01:34:32] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Better "Maximum impressions" any individual will see - https://phabricator.wikimedia.org/T421662#11762621 (10Krinkle) [01:38:08] RECOVERY - check_load on franio2002 is OK: LOAD OK - total load average: 0.37, 2.77, 9.54 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2002&service=check_load [01:38:08] RECOVERY - check_load on franio2001 is OK: LOAD OK - total load average: 0.85, 3.76, 8.57 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2001&service=check_load [01:41:50] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Better "Maximum impressions" any individual will see - https://phabricator.wikimedia.org/T421662#11762624 (10Krinkle) Notes from brainstorming session between @Ejegg and myself (26 Feb 2026) on various goals under T417064 including this one: > Goal... [02:54:49] (03PS1) 10Krinkle: BannerHistoryLogger: Remove unused `waitLogNoSendBeacon` mixin param [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1264185 (https://phabricator.wikimedia.org/T421662) [02:58:02] (03CR) 10CI reject: [V:04-1] BannerHistoryLogger: Remove unused `waitLogNoSendBeacon` mixin param [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1264185 (https://phabricator.wikimedia.org/T421662) (owner: 10Krinkle) [03:10:15] (03PS2) 10Krinkle: BannerHistoryLogger: Remove unused `waitLogNoSendBeacon` mixin param [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1264185 (https://phabricator.wikimedia.org/T421662) [05:34:12] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog, 07fr-current-sprint, 05MW-1.46-notes (1.46.0-wmf.22; 2026-03-31), 07payments-orchestration: Add VIPPS in Norway - https://phabricator.wikimedia.org/T390870#11762764 (10RKumar_WMF) Please provide the test link once it is ready. Ty! [06:03:45] 06Fundraising-Backlog, 06FR-donorrelations: Outreach to the donors from T420651 - https://phabricator.wikimedia.org/T421534#11762788 (10krobinson) [[ https://docs.google.com/document/d/1QJG1vYLqrlDgXS1xI8KFI_osIXL17ONKG-uVkXV1R-s/edit?tab=t.0 | This is the email copy document ]] we used last time. Note that w... [06:11:16] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 96% used. Largest process: trino-server-co (1564) = 95.4% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [06:16:14] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 96% used. Largest process: trino-server-co (1564) = 95.4% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [06:21:14] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 96% used. Largest process: trino-server-co (1564) = 95.4% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [06:31:14] RECOVERY - check_memory on fransw2003 is OK: OK Memory 86% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [13:53:31] 10fundraising-tech-ops, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install frdb1008 - https://phabricator.wikimedia.org/T414374#11765358 (10Jgreen) a:05Jgreen→03VRiley-WMF @VRiley-WMF I'm not having much luck with this box. Running into two more issues: - iDRAC not handling terminal correctly, arrow... [14:14:22] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog, 07fr-current-sprint, 05MW-1.46-notes (1.46.0-wmf.22; 2026-03-31), 07payments-orchestration: Add VIPPS in Norway - https://phabricator.wikimedia.org/T390870#11765457 (10Ejegg) @RKumar_WMF this link will at least get us a redirect to the Vipps fo... [14:31:40] 10Fundraising analytics stack: Ingest Comscore data into the data lakehouse - https://phabricator.wikimedia.org/T417312#11765590 (10IAckerman-WMF) @soworu will pull the full dataset in csv format on a monthly basis. [14:31:52] 10Fundraising analytics stack: Ingest Comscore data into the data lakehouse - https://phabricator.wikimedia.org/T417312#11765592 (10IAckerman-WMF) (Thank you!) [14:32:13] PROBLEM - check_load on fransw2002 is CRITICAL: LOAD CRITICAL - total load average: 71.42, 65.54, 51.81 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [14:37:11] RECOVERY - check_load on fransw2002 is OK: LOAD OK - total load average: 25.63, 45.13, 47.29 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [14:41:11] (03PS1) 10Ejegg: Fix Vipps IPN handling [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1264646 (https://phabricator.wikimedia.org/T390870) [14:41:45] (03CR) 10CI reject: [V:04-1] Fix Vipps IPN handling [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1264646 (https://phabricator.wikimedia.org/T390870) (owner: 10Ejegg) [14:42:50] (03PS2) 10Ejegg: Fix Vipps IPN handling [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1264646 (https://phabricator.wikimedia.org/T390870) [14:43:10] damilare|brb: oops, turns out it needed a small code change too, not just config ^^^^ [14:44:00] (03CR) 10Damilare Adedoyin: [C:03+2] Fix Vipps IPN handling [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1264646 (https://phabricator.wikimedia.org/T390870) (owner: 10Ejegg) [14:44:03] thanks! [14:44:14] I'll get that out to standalone smashpig [14:44:17] np [14:44:31] (03Merged) 10jenkins-bot: Fix Vipps IPN handling [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1264646 (https://phabricator.wikimedia.org/T390870) (owner: 10Ejegg) [14:44:55] ah, just noticed i have a lying copypasted @covers annotation there [14:44:58] oh well [14:48:05] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1264650 [14:48:07] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1264650 (owner: 10Ejegg) [14:48:33] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donor-portal: Add frequency, native currency amount and USD amount to recurring pause and cancel activities from Donor Portal - https://phabricator.wikimedia.org/T421733 (10Lars) 03NEW [14:48:36] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1264650 (owner: 10Ejegg) [14:49:09] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add frequency, native currency amount and USD amount to recurring pause and cancel activities from DR action - https://phabricator.wikimedia.org/T421734 (10Lars) 03NEW [14:49:45] yep, that should be easy enough larssandergreen [14:50:31] I was hoping that was what you were going to say [14:51:35] !log standalone (IPN listener) SmashPig upgraded from 545f0b10 to abf8682a [14:51:37] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [15:31:21] (03PS1) 10Ejegg: Pending message lookup defaults to including resolved [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1264664 (https://phabricator.wikimedia.org/T421282) [15:31:53] (03CR) 10CI reject: [V:04-1] Pending message lookup defaults to including resolved [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1264664 (https://phabricator.wikimedia.org/T421282) (owner: 10Ejegg) [15:36:13] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations, and 2 others: Add global Do not contact button to contact summary - https://phabricator.wikimedia.org/T404989#11766104 (10krobinson) @Lars do we have an update on this one? Ideally we would have this li... [15:36:33] (03PS2) 10Ejegg: Pending message lookup defaults to including resolved [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1264664 (https://phabricator.wikimedia.org/T421282) [15:37:04] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog, 07fr-current-sprint, 05MW-1.46-notes (1.46.0-wmf.22; 2026-03-31), 07payments-orchestration: Add VIPPS in Norway - https://phabricator.wikimedia.org/T390870#11766107 (10RKumar_WMF) thank you. Will share with the tester. [15:37:34] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations, and 2 others: Add global Do not contact button to contact summary - https://phabricator.wikimedia.org/T404989#11766109 (10Lars) I think ideally we'd do this in conjunction with the consent matrix change... [15:45:27] (03CR) 10Lars SG: [C:03+2] See if this bubbles up deprecations [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260827 (owner: 10Eileen) [15:47:47] (03CR) 10Lars SG: [C:03+2] Early return if no amount is set [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260236 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [16:03:39] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 05MW-1.46-notes (1.46.0-wmf.22; 2026-03-31): PayPal recurring fail wellness check - https://phabricator.wikimedia.org/T420651#11766294 (10Ejegg) Here's a searchkit to find all the people affected with a cancel_date after Feb 28th:... [16:03:54] 06Fundraising-Backlog, 06FR-donorrelations: Outreach to the donors from T420651 - https://phabricator.wikimedia.org/T421534#11766296 (10Ejegg) Here's a searchkit to find all the people affected with a cancel_date after Feb 28th: https://civicrm.wikimedia.org/civicrm/search#/display/Recurrings_cancelled_from_pa... [16:14:10] (03CR) 10CI reject: [V:04-1] See if this bubbles up deprecations [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260827 (owner: 10Eileen) [16:37:40] (03CR) 10CI reject: [V:04-1] Early return if no amount is set [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260236 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [16:38:17] (03PS1) 10Ejegg: Add gateway txn hints to pending message lookups [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1264678 [16:42:09] 10fundraising-tech-ops, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install frdata1003, frmx1002, frqueue100[5-6] - https://phabricator.wikimedia.org/T416249#11766443 (10Jgreen) All four are switched to UEFI and built. [16:59:53] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add frequency, native currency amount and USD amount to recurring pause and cancel activities from DR action - https://phabricator.wikimedia.org/T421734#11766550 (10Lars) a:03Lars [17:02:43] 06Fundraising-Backlog, 06FR-donorrelations: Outreach to the donors from T420651 - https://phabricator.wikimedia.org/T421534#11766563 (10SHust) That would be great to add, @Ejegg, thank you! [17:14:46] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: pfw-eqiad NAT for frmx1002.wikimedia.org - https://phabricator.wikimedia.org/T421750 (10Jgreen) 03NEW [17:29:39] 06Fundraising-Backlog: Allow SFTP outbound requests from civi1002 to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T421751 (10jgleeson) 03NEW [17:30:02] 06Fundraising-Backlog: Allow SFTP outbound requests from civi2001 to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T421751#11766698 (10jgleeson) [17:42:54] 06Fundraising-Backlog: Allow SFTP outbound requests from civi2001 to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T421751#11766772 (10jgleeson) [17:43:16] dwisehaupt: Jeff_Green ^ [17:59:43] 06Fundraising-Backlog, 07fr-payments-ACH, 07payments-orchestration: Rescue ach recurring expired token issue to avoid charge back fee - https://phabricator.wikimedia.org/T421756 (10AnnWF) 03NEW [18:02:21] jgleeson|food: thanks. [18:03:27] ejegg: https://teams.microsoft.com/meet/2535222302037?p=VIYG6245R7krjc4Ob1 [18:06:09] (03PS1) 10Lars SG: Add frequency, currency, native currency amount and USD amount to recurring pause and cancel activities from Donor Portal [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264703 (https://phabricator.wikimedia.org/T421733) [18:08:00] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donor-portal: Add frequency, native currency amount and USD amount to recurring pause and cancel activities from Donor Portal - https://phabricator.wikimedia.org/T421733#11766938 (10Lars) [18:08:37] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add frequency, native currency amount and USD amount to recurring pause and cancel activities from DR action - https://phabricator.wikimedia.org/T421734#11766942 (10Lars) [18:30:50] (03CR) 10CI reject: [V:04-1] Add frequency, currency, native currency amount and USD amount to recurring pause and cancel activities from Donor Portal [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264703 (https://phabricator.wikimedia.org/T421733) (owner: 10Lars SG) [19:18:32] (03PS1) 10Ejegg: Test for mapping Gravy buyer ID validation error [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1264722 (https://phabricator.wikimedia.org/T421155) [19:19:07] (03CR) 10CI reject: [V:04-1] Test for mapping Gravy buyer ID validation error [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1264722 (https://phabricator.wikimedia.org/T421155) (owner: 10Ejegg) [19:20:37] (03CR) 10Ejegg: [C:04-1] "Let's try to fix this at the SmashPig level instead of having to repeat the handling here for misplaced validation errors. I think if we c" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1262356 (https://phabricator.wikimedia.org/T421155) (owner: 10Wfan) [19:21:31] ejegg, if we do from smashpig level, this message still gonna skipped since it contains rawResponse [19:22:22] do you mind to keep the failmail even if line 119? [19:23:24] ahhh i see [19:23:38] ok, so we probably want to failmail even if there is a rawResponse, huh? [19:24:06] Sorry, I guess we DO want to make that change at least [19:24:14] but I don't think we need to go through both arrays [19:24:22] I think so, but still need to map this error to validation since it do not have error_code [19:26:04] in our current logic if we want to get failmail we should have ['raw_response']['error_code'] in array of cvv or tax identifier [19:26:29] we will need to go over both array if you check addPaymentFailureError [19:28:00] we only assign error to validation error if error_code is exist and about cvv expiration date and tax id, others all categorized to general Error, so the one I had is check general error if matched validation error then return failmail [19:41:41] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog: Allow SFTP outbound requests from civi2001 to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T421751#11767367 (10jgleeson) [19:41:56] (03PS1) 10Ejegg: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1264729 [19:41:59] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1264729 (owner: 10Ejegg) [19:46:10] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog: Allow SFTP outbound requests from civi2001 to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T421751#11767391 (10SBurnett-WMF) Dallas will take a run at it [19:47:09] (03Merged) 10jenkins-bot: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1264729 (owner: 10Ejegg) [19:47:22] (03PS3) 10Eileen: Fix is_major_gift for subsequent recurrings [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260198 (https://phabricator.wikimedia.org/T409994) [19:47:55] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog, 10fundraising-tech-ops: Allow SFTP outbound requests from civi2001 to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T421751#11767395 (10Dwisehaupt) p:05Triage→03High [19:48:51] (03PS4) 10Eileen: See if this bubbles up deprecations [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260827 [19:49:31] (03PS2) 10Eileen: Early return if no amount is set [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260236 (https://phabricator.wikimedia.org/T409994) [19:49:56] (03PS3) 10Eileen: Update is_major_gift based on utm_medium [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260304 (https://phabricator.wikimedia.org/T409994) [19:50:18] (03PS7) 10Eileen: Initial Stripe Audit Processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1252306 (https://phabricator.wikimedia.org/T419628) [19:51:37] 03Fundraising Sprint - Floor is Lava, 07fr-current-sprint, 05FY25-26 WE3.5 Donor Identification and recognition: Add SFTP upload support to MediaWiki donor export job - https://phabricator.wikimedia.org/T421772#11767425 (10jgleeson) [19:52:35] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Add fields to Civi to better reflect reporting -GL Code - https://phabricator.wikimedia.org/T420156#11767431 (10SBurnett-WMF) [20:00:47] 06Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 07fr-current-sprint: Refactor session use in DonationInterface - https://phabricator.wikimedia.org/T417831#11767456 (10SBurnett-WMF) [20:02:24] 06Fundraising-Backlog, 10fundraising-tech-ops, 07fr-current-sprint, 07FR-Fraud: Create a payment attempts table for fraud scoring service - https://phabricator.wikimedia.org/T417217#11767460 (10SBurnett-WMF) [20:02:40] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Fix testRecurringQueueConsumeAutoRescueMessage when previous month lacks numeric day of current month - https://phabricator.wikimedia.org/T421773 (10Lars) 03NEW [20:06:02] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260198 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [20:06:13] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260304 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [20:06:22] (03PS1) 10Lars SG: Fix testRecurringQueueConsumeAutoRescueMessage when previous month lacks numeric day of current month [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264734 (https://phabricator.wikimedia.org/T421773) [20:06:44] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260827 (owner: 10Eileen) [20:06:59] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260236 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [20:07:56] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Add contribution search Searchkit/Formbuilder for DR - https://phabricator.wikimedia.org/T404987#11767496 (10Lars) [20:09:49] 06Fundraising-Backlog, 06FR-donorrelations, 07fr-current-sprint: Feature request assessment: Zendesk macro updater - https://phabricator.wikimedia.org/T387503#11767511 (10SBurnett-WMF) [20:10:48] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations, 07fr-current-sprint: Improve automatic deduping - https://phabricator.wikimedia.org/T415806#11767514 (10Lars) a:03Lars [20:11:08] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donor-portal: Add frequency, native currency amount and USD amount to recurring pause and cancel activities from Donor Portal - https://phabricator.wikimedia.org/T421733#11767518 (10Lars) a:03Lars [20:14:10] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Fix testRecurringQueueConsumeAutoRescueMessage when previous month lacks numeric day of current month - https://phabricator.wikimedia.org/T421773#11767524 (10Lars) a:03Lars [20:14:32] (03CR) 10CI reject: [V:04-1] Initial Stripe Audit Processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1252306 (https://phabricator.wikimedia.org/T419628) (owner: 10Eileen) [20:16:13] (03PS2) 10Eileen: Treat all 'civi_mail%' utm_source as channel email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260561 (https://phabricator.wikimedia.org/T409994) [20:16:21] (03PS2) 10Eileen: Update is_major_gift where utm_source like ~MG [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260562 (https://phabricator.wikimedia.org/T409994) [20:17:53] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations, and 2 others: Epic: Civi much slower than usual! - https://phabricator.wikimedia.org/T402837#11767530 (10Lars) [20:17:58] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations, and 2 others: Epic: Civi much slower than usual! - https://phabricator.wikimedia.org/T402837#11767533 (10Lars) a:03Lars [20:19:57] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Repair contribution tracking records where utm_medium overwritten by 'Audit' - https://phabricator.wikimedia.org/T421035#11767549 (10SBurnett-WMF) p:05Triage→03High [20:21:47] (03CR) 10CI reject: [V:04-1] Update is_major_gift based on utm_medium [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260304 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [20:21:49] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Fix older Adyen Gr4vy / gravy backend_gateway_txn_id - https://phabricator.wikimedia.org/T421034#11767555 (10SBurnett-WMF) p:05Triage→03High the mismatched IDs aren't fixed yet but this may coincide... [20:21:54] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Investigate inconsistent email display during record merges - https://phabricator.wikimedia.org/T418320#11767557 (10Lars) No one in Civi-land has objected (and we had one vote in support), so I thi... [20:22:17] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Investigate inconsistent email display during record merges - https://phabricator.wikimedia.org/T418320#11767559 (10Lars) a:05Lars→03None [20:22:37] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Backfill location type for emails for existing contacts using Apple, Google, Amazon, Venmo or PayPal - https://phabricator.wikimedia.org/T420992#11767561 (10SBurnett-WMF) p:05Triage→03Medium [20:23:20] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Investigate why queries often seem to go to the rw database instead of the read only replica - https://phabricator.wikimedia.org/T420986#11767567 (10SBurnett-WMF) p:05Triage→03Medium [20:25:43] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: High priority - audit overwriting contribution tracking - https://phabricator.wikimedia.org/T421024#11767570 (10Lars) p:05Triage→03High [20:26:05] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Trustly donations stuck at authorized - https://phabricator.wikimedia.org/T404446#11767572 (10Damilare) p:05Medium→03Low [20:26:11] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Trustly donations stuck at authorized - https://phabricator.wikimedia.org/T404446#11767574 (10Damilare) p:05Low→03Medium [20:26:59] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fr-tech chores list - https://phabricator.wikimedia.org/T258527#11767579 (10XenoRyet) p:05Triage→03High [20:27:01] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fr-tech chores list - https://phabricator.wikimedia.org/T258527#11767582 (10XenoRyet) p:05High→03Triage [20:27:03] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Smashpig, 10Recurring-Donations, 13Patch-For-Review: Ensure validation errors in recurring charges send failmail - https://phabricator.wikimedia.org/T421155#11767581 (10AnnWF) p:05Triage→03Medium [20:27:10] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donor-portal, 13Patch-For-Review: Donor Portal - Wikiminute video issues on mobile devices - https://phabricator.wikimedia.org/T420782#11767584 (10AnnWF) p:05Triage→03Medium [20:31:31] (03CR) 10CI reject: [V:04-1] Early return if no amount is set [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260236 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [20:32:04] (03CR) 10CI reject: [V:04-1] Fix is_major_gift for subsequent recurrings [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260198 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [20:33:43] (03CR) 10CI reject: [V:04-1] See if this bubbles up deprecations [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260827 (owner: 10Eileen) [20:37:00] (03CR) 10Eileen: [V:03+2] "Gonna v+2 this as it is unrelated date issue causing fail" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260198 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [20:38:17] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog, 10fundraising-tech-ops, 06ServiceOps new: Allow SFTP outbound requests from civi2001 to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T421751#11767630 (10Dwisehaupt) a:05Jgreen→03Dwisehaupt iptables rules are up and working on th... [20:38:22] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1264737 [20:39:25] (03CR) 10CI reject: [V:04-1] Treat all 'civi_mail%' utm_source as channel email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260561 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [20:39:26] (03CR) 10Lars SG: [C:03+2] Update is_major_gift based on utm_medium [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260304 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [20:39:33] (03CR) 10Eileen: [C:03+2] Fix testRecurringQueueConsumeAutoRescueMessage when previous month lacks numeric day of current month [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264734 (https://phabricator.wikimedia.org/T421773) (owner: 10Lars SG) [20:39:49] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1264737 (owner: 10Eileen) [20:40:26] (03CR) 10CI reject: [V:04-1] Update is_major_gift where utm_source like ~MG [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260562 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [20:40:35] (03CR) 10Lars SG: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264703 (https://phabricator.wikimedia.org/T421733) (owner: 10Lars SG) [20:40:43] !log civicrm upgraded from 58426b1e to 3724cc2d [20:40:44] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [20:43:01] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add recipient to $10k+ Large Donation Bot emails - https://phabricator.wikimedia.org/T421018#11767671 (10Ejegg) Just added [20:44:45] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog, 10fundraising-tech-ops, 06ServiceOps new: Allow SFTP outbound requests from civi1002/civi2002 to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T421751#11767676 (10Dwisehaupt) [20:45:37] back in a bit... [20:48:51] thanks dwisehaupt for fixing that [20:50:47] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Spike: Add monitoring for transactional emails - https://phabricator.wikimedia.org/T421779 (10Lars) 03NEW [20:55:53] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Replace HEPdata employer matching data with something that updates a little more often - https://phabricator.wikimedia.org/T421296#11767751 (10Lars) The plan is to replace HEPData with another provider with a more sophisticated solution once ComboWiki has... [20:56:31] 03Fundraising Sprint - Floor is Lava, 06Fundraising-Backlog, 10fundraising-tech-ops, 06ServiceOps new: Allow SFTP outbound requests from civi1002/civi2002 to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T421751#11767752 (10CDanis) Apologies, I'll write a host firewall patch tomorrow morning. [20:58:39] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1252306 (https://phabricator.wikimedia.org/T419628) (owner: 10Eileen) [21:02:41] (03CR) 10CI reject: [V:04-1] Update is_major_gift based on utm_medium [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260304 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [21:05:10] (03CR) 10Eileen: [V:03+2] "unrelated fail" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260304 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [21:06:18] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1264745 [21:06:25] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1264745 (owner: 10Eileen) [21:07:24] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1264745 (owner: 10Eileen) [21:08:13] (03PS3) 10Eileen: Treat all 'civi_mail%' utm_source as channel email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260561 (https://phabricator.wikimedia.org/T409994) [21:08:29] (03CR) 10CI reject: [V:04-1] Treat all 'civi_mail%' utm_source as channel email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260561 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [21:08:45] (03PS4) 10Eileen: Treat all 'civi_mail%' utm_source as channel email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260561 (https://phabricator.wikimedia.org/T409994) [21:08:53] (03CR) 10Lars SG: [C:03+2] Treat all 'civi_mail%' utm_source as channel email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260561 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [21:09:25] !log civicrm upgraded from 3724cc2d to 7d299b48 [21:09:26] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [21:14:31] (03PS1) 10Eileen: Fix c.appeal to v.appeal [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264747 [21:14:44] larssandergreen: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1264747 [21:15:10] (03CR) 10Wfan: Test for mapping Gravy buyer ID validation error (031 comment) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1264722 (https://phabricator.wikimedia.org/T421155) (owner: 10Ejegg) [21:16:26] (03CR) 10Lars SG: [C:03+2] Fix c.appeal to v.appeal [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264747 (owner: 10Eileen) [21:17:27] (03PS2) 10Wfan: Test for mapping Gravy buyer ID validation error [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1264722 (https://phabricator.wikimedia.org/T421155) (owner: 10Ejegg) [21:17:54] (03CR) 10CI reject: [V:04-1] Test for mapping Gravy buyer ID validation error [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1264722 (https://phabricator.wikimedia.org/T421155) (owner: 10Ejegg) [21:21:04] 03Fundraising Sprint - Floor is Lava, 07fr-current-sprint, 05FY25-26 WE3.5 Donor Identification and recognition: Add SFTP upload support to MediaWiki donor export job - https://phabricator.wikimedia.org/T421772#11767832 (10Jdlrobson-WMF) p:05Triage→03High [21:34:53] (03PS3) 10Wfan: Test for mapping Gravy buyer ID validation error [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1264722 (https://phabricator.wikimedia.org/T421155) (owner: 10Ejegg) [21:35:31] 03Fundraising Sprint - Floor is Lava, 14Fundraising Sprint: Vixen, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 3 others: Ensure new is_major_gift & gl code fields are fully and correctly populated - https://phabricator.wikimedia.org/T409994#11767864 (10MDemosWMF) @Eileenmcnaughton I am not... [21:36:25] ejegg: for validation error failmail, Which one do you like the most? 1: add all validation error without error_code to a default error code like"other_validation_error" then failmail if any validation Errors no matter raw response exist or not 2: current patch to loop both error and validation error. if error with code validation then failmail? [21:37:22] (03Merged) 10jenkins-bot: Treat all 'civi_mail%' utm_source as channel email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260561 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [21:40:39] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10FR-Donor-portal, 06FR-donorrelations, and 2 others: PDF Civi receipts - feature request - https://phabricator.wikimedia.org/T368671#11767868 (10MDemosWMF) @Lars @AKanji-WMF Some donors request a one-off TYL to be mailed to them or emailed in a PDF format... [21:43:33] (03CR) 10Lars SG: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1260562 (https://phabricator.wikimedia.org/T409994) (owner: 10Eileen) [21:43:34] (03Merged) 10jenkins-bot: Fix c.appeal to v.appeal [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264747 (owner: 10Eileen) [21:47:20] (03PS3) 10Jgleeson: Add optional AES encryption for mediawiki donor export output [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1253624 (https://phabricator.wikimedia.org/T416948) [21:47:20] (03PS2) 10Jgleeson: Limit MW donor export CSV to email and donor_status_id only [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1256389 (https://phabricator.wikimedia.org/T416948) [21:47:20] (03PS1) 10Jgleeson: Add optional SFTP upload for mediawiki donor export [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1264753 (https://phabricator.wikimedia.org/T421772) [21:48:13] (03CR) 10CI reject: [V:04-1] Add optional SFTP upload for mediawiki donor export [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1264753 (https://phabricator.wikimedia.org/T421772) (owner: 10Jgleeson) [21:53:59] (03PS2) 10Jgleeson: Add optional SFTP upload for mediawiki donor export [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1264753 (https://phabricator.wikimedia.org/T421772) [22:01:45] 03Fundraising Sprint - Floor is Lava, 07fr-current-sprint, 05FY25-26 WE3.5 Donor Identification and recognition: Create SFTP keypair for MediaWiki donor export - https://phabricator.wikimedia.org/T421785 (10jgleeson) 03NEW [22:02:35] 03Fundraising Sprint - Floor is Lava, 07fr-current-sprint, 05FY25-26 WE3.5 Donor Identification and recognition: Create SFTP keypair for MediaWiki donor export - https://phabricator.wikimedia.org/T421785#11767996 (10jgleeson) Once complete, we need to send the public key over to @CDanis [22:03:18] (03PS1) 10Eileen: Joseph changed his mind [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264760 [22:10:51] (03CR) 10Lars SG: [C:03+2] Joseph changed his mind [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264760 (owner: 10Eileen) [22:15:29] (03CR) 10Lars SG: [C:03+2] Add test for braintree chargeback_reversal [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1261664 (owner: 10Eileen) [22:21:28] 03Fundraising Sprint - Floor is Lava, 07fr-current-sprint, 05FY25-26 WE3.5 Donor Identification and recognition: Create SFTP keypair for MediaWiki donor export - https://phabricator.wikimedia.org/T421785#11768071 (10Dwisehaupt) @CDanis Here is the pubkey we will be using. I'll work with Jack to get it in our... [22:38:37] (03Merged) 10jenkins-bot: Joseph changed his mind [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264760 (owner: 10Eileen) [22:38:39] (03Merged) 10jenkins-bot: Add test for braintree chargeback_reversal [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1261664 (owner: 10Eileen) [22:57:34] (03PS1) 10Eileen: Update our offline gifts to have offline for channel, is_major_gift = TRUE. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264780 (https://phabricator.wikimedia.org/T409994) [23:02:33] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1264784 [23:02:37] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1264785 [23:02:53] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1264784 (owner: 10Eileen) [23:03:09] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1264785 (owner: 10Eileen) [23:04:14] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1264784 (owner: 10Eileen) [23:04:22] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1264785 (owner: 10Eileen) [23:19:26] !log civicrm upgraded from 7d299b48 to cf871dd3 [23:19:27] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [23:23:00] (03PS2) 10Eileen: Update our offline gifts to have offline for channel, is_major_gift = TRUE. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264780 (https://phabricator.wikimedia.org/T409994) [23:23:04] (03PS1) 10Eileen: Set channel = Other Online For remaining online gifts [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264797 (https://phabricator.wikimedia.org/T409994)