[00:05:28] (03PS2) 10Wfan: Add consent if email update from EPC [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1196541 (https://phabricator.wikimedia.org/T407398) [00:07:36] (03PS6) 10Wfan: Add more lowConfidence payment method sources [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196140 (https://phabricator.wikimedia.org/T406153) [00:09:07] (03CR) 10Cstone: [C:03+2] Do not update contribution fields on settle for negative transactions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196225 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [00:09:54] (03CR) 10Cstone: [C:03+2] Fix handling of negative fee [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1196394 (https://phabricator.wikimedia.org/T406684) (owner: 10Eileen) [00:10:39] (03Merged) 10jenkins-bot: Fix handling of negative fee [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1196394 (https://phabricator.wikimedia.org/T406684) (owner: 10Eileen) [00:11:53] (03PS6) 10Eileen: Fix net vs total on transactions [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1196525 (https://phabricator.wikimedia.org/T152249) [00:12:06] (03CR) 10CI reject: [V:04-1] Fix net vs total on transactions [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1196525 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [00:12:37] (03PS7) 10Eileen: Fix net vs total on transactions [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1196525 (https://phabricator.wikimedia.org/T152249) [00:24:57] (03PS8) 10Eileen: Fix net vs total on transactions [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1196525 (https://phabricator.wikimedia.org/T152249) [00:26:43] (03Merged) 10jenkins-bot: Do not update contribution fields on settle for negative transactions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196225 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [00:28:13] (03CR) 10CI reject: [V:04-1] Add more lowConfidence payment method sources [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196140 (https://phabricator.wikimedia.org/T406153) (owner: 10Wfan) [00:30:05] (03PS2) 10Eileen: Fixes to do a batch test with payout validation [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196538 (https://phabricator.wikimedia.org/T152249) [00:45:57] (03CR) 10CI reject: [V:04-1] Fixes to do a batch test with payout validation [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196538 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [00:48:07] (03CR) 10Cstone: [C:03+2] Fix net vs total on transactions [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1196525 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [00:48:36] (03Merged) 10jenkins-bot: Fix net vs total on transactions [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1196525 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [01:03:14] cstone: did you tag smashpig? [01:03:39] No right that merges fast doesn't it [01:03:45] Want me to tag it with those new ones ? [01:04:21] yep please I think that's it ... for now [01:04:39] Okie [01:10:27] (03PS3) 10Eileen: Record fees when they come in [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195824 (https://phabricator.wikimedia.org/T152249) [01:13:02] (03PS1) 10Cstone: Update smash-pig to v0.9.11.5 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196545 [01:13:32] (03CR) 10Cstone: [C:03+2] Update smash-pig to v0.9.11.5 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196545 (owner: 10Cstone) [01:13:48] (03PS3) 10Eileen: Fixes to do a batch test with payout validation [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196538 (https://phabricator.wikimedia.org/T152249) [01:13:48] (03PS4) 10Eileen: Record fees when they come in [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195824 (https://phabricator.wikimedia.org/T152249) [01:15:12] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /srv 29728MiB (10% inode=97%): /dev 31930MiB (100% inode=99%): /run 6390MiB (99% inode=99%): / 3304MiB (47% inode=79%): /dev/shm 31960MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 290MiB (68% inode=99%): /home 11068MiB (89% inode=99%): /tmp 10383MiB (84% inode=98%): /var 19273MiB (72% inode=83%): /var/tmp 422MiB (99% inode=99%): /var/log 15046MiB (56 [01:15:12] 9%): /var/log/audit 422MiB (99% inode=99%): /run/user/12807 6392MiB (100% inode=99%): /run/user/16759 6392MiB (100% inode=99%): /run/user/16765 6392MiB (100% inode=99%): /run/user/3649 6392MiB (100% inode=99%): /run/user/33 6392MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [01:16:14] hmmm [01:16:38] (03PS1) 10Wfan: Add more lowConfidence payment method sources [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196546 (https://phabricator.wikimedia.org/T406153) [01:17:59] (03PS7) 10Wfan: Add more lowConfidence payment method sources [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196140 (https://phabricator.wikimedia.org/T406153) [01:20:10] RECOVERY - check_disk on civi1002 is OK: DISK OK - free space: /srv 48684MiB (16% inode=97%): /dev 31930MiB (100% inode=99%): /run 6390MiB (99% inode=99%): / 3304MiB (47% inode=79%): /dev/shm 31960MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 290MiB (68% inode=99%): /home 11068MiB (89% inode=99%): /tmp 10383MiB (84% inode=98%): /var 19273MiB (72% inode=83%): /var/tmp 422MiB (99% inode=99%): /var/log 15002MiB (56% inode=99% [01:20:10] og/audit 422MiB (99% inode=99%): /run/user/12807 6392MiB (100% inode=99%): /run/user/16759 6392MiB (100% inode=99%): /run/user/16765 6392MiB (100% inode=99%): /run/user/3649 6392MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [01:22:06] ^ i deleted some audit files that werent ever going to be run if that did fix that [01:22:35] (03Abandoned) 10Eileen: wip [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196195 (owner: 10Eileen) [01:26:57] (03CR) 10CI reject: [V:04-1] Record fees when they come in [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195824 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [01:28:36] (03CR) 10CI reject: [V:04-1] Update smash-pig to v0.9.11.5 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196545 (owner: 10Cstone) [01:32:32] cstone: we ran outta space?? [01:32:38] or near... [01:32:47] there are log files that can go too I think [01:32:57] cstone: smashpig passes with this - https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1196538/3 [01:34:04] i think it was more 10% was left I read it right [01:34:14] and then when i deleted the other files it went to 16% [01:37:33] lul civicrm.wmf.INFO: Batch total mismatch. USD is out by 1735634.96. Expected 1707093.30 vs Actual -28541.66 [01:43:27] (03CR) 10Cstone: [C:03+2] "the numbers add up!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196538 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [01:44:49] lol - opps that's a gap [01:45:23] (03CR) 10Eileen: [V:03+2] Update smash-pig to v0.9.11.5 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196545 (owner: 10Cstone) [01:45:25] (03CR) 10Eileen: [V:03+2 C:03+2] Update smash-pig to v0.9.11.5 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196545 (owner: 10Cstone) [01:45:42] well ignoring that total amount all the fees and such added up [01:55:03] cstone: which total amount ... [01:55:11] my bad one [01:55:14] oh ok [01:55:16] they dont add up in my example hah [01:55:21] but the fees all do [01:56:14] what do the @s do in the save for the dates? [01:56:55] oh - they make it see a number as a timestamp [01:57:14] strtotime [01:57:30] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Donors reporting trouble with credit card fields using Firefox - https://phabricator.wikimedia.org/T404645#11279723 (10KHancock99) @jgleeson It it helps the Phab investigation and a replication attempt in any way, I've just had a do... [01:57:32] I'm trying to figure out where I left my remaining work... [01:57:46] ahh [01:57:48] fancy [01:59:41] (03PS14) 10Eileen: Add generateBatch api + test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1193485 (https://phabricator.wikimedia.org/T396071) [01:59:41] (03PS6) 10Eileen: Do not merge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1194132 [01:59:41] (03PS6) 10Eileen: Do not merge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1194154 [02:03:40] eileen: its still inserting them but throwing civicrm.wmf.INFO: wmf_civicrm: Failed to parse trxn_id: ADYEN Transaction Fees October 11 2025, INVALID_MESSAGE Malformed unique id (too many terms): ADYEN Transaction Fees October 11 2025 [02:04:03] cstone: where is it inserting them? into the queue? [02:04:14] no into the db for the fees [02:04:19] its just giving that line [02:04:22] info line [02:04:36] Oh - so working but error cos it's trying twice? [02:05:04] no it can handle trying twice fine [02:05:13] its on the insert itself its falling into that [02:05:28] im guessing cuase its just expecting ADYEN 1234 [02:05:35] theres a bunch of cases where that error is [02:06:20] and did you want total amount to be 0? [02:06:48] yeah - I landed on 0 in the end cos no-one pays us in a donor sorta way [02:07:00] okie as long as its what you expect [02:07:11] yeah - no right answer on that one [02:07:42] i think for this warning we just need to add the fee to the unique id thing can do that later [02:08:40] (03CR) 10Cstone: [C:03+2] Record fees when they come in [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195824 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [02:15:10] oh ok I get that cstone - I have one more thing to sort - which is bring my chargeback reversed back to life & then in theory.... [02:15:25] I should probably empty the smashpig settle queue too & try to re-fill it [02:15:42] (03CR) 10CI reject: [V:04-1] Do not merge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1194154 (owner: 10Eileen) [02:16:38] (03CR) 10CI reject: [V:04-1] Add generateBatch api + test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1193485 (https://phabricator.wikimedia.org/T396071) (owner: 10Eileen) [02:18:15] (03CR) 10CI reject: [V:04-1] Do not merge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1194132 (owner: 10Eileen) [02:19:39] (03Merged) 10jenkins-bot: Fixes to do a batch test with payout validation [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196538 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [02:26:40] (03Merged) 10jenkins-bot: Record fees when they come in [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195824 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [02:31:39] (03PS1) 10Eileen: Fix Contribution Tracking to join better to contribution in SK [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196547 [02:31:58] ^^ should be an easy +2 [02:35:24] (03PS1) 10Eileen: Upgrade wikimedia/smash-pig to v0.9.11.5 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1196548 [02:35:44] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1196549 [02:35:57] (03CR) 10Eileen: [C:03+2] Upgrade wikimedia/smash-pig to v0.9.11.5 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1196548 (owner: 10Eileen) [02:36:10] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1196549 (owner: 10Eileen) [02:40:21] (03PS15) 10Eileen: Add generateBatch api + test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1193485 (https://phabricator.wikimedia.org/T396071) [02:40:21] (03PS7) 10Eileen: Do not merge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1194132 [02:40:21] (03PS7) 10Eileen: Do not merge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1194154 [02:48:04] (03CR) 10Cstone: [C:03+2] Fix Contribution Tracking to join better to contribution in SK [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196547 (owner: 10Eileen) [02:49:59] !log civicrm upgraded from 25df5996 to ac4c185b [02:50:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:58:47] (03CR) 10CI reject: [V:04-1] Do not merge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1194154 (owner: 10Eileen) [02:59:21] (03CR) 10CI reject: [V:04-1] Add generateBatch api + test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1193485 (https://phabricator.wikimedia.org/T396071) (owner: 10Eileen) [03:00:23] (03CR) 10CI reject: [V:04-1] Do not merge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1194132 (owner: 10Eileen) [03:43:29] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: phab test 55 - https://phabricator.wikimedia.org/T407459 (10AKanji-WMF) 03NEW p:05Triage→03Low [03:51:31] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: phab test 55 - https://phabricator.wikimedia.org/T407459#11279779 (10AKanji-WMF) 05Open→03Invalid [04:38:21] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Donors reporting trouble with credit card fields using Firefox - https://phabricator.wikimedia.org/T404645#11279819 (10KHancock99) @jgleeson For the love of the Phab gods, I hope this is the very last ping from me on this because **... [05:52:15] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07FR-Audience-Dev: Annual Recurring Statuses implementation in Civi (Audience Framework update) - https://phabricator.wikimedia.org/T377438#11279915 (10MSuijkerbuijk_WMF) @Lars I'd say we should confirm that point - please have a look at how it was set... [07:10:46] (03PS1) 10Eileen: Record chargebackreversals when received [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196563 (https://phabricator.wikimedia.org/T157456) [07:11:16] (03PS2) 10Eileen: Record chargebackreversals when received [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196563 (https://phabricator.wikimedia.org/T157456) [09:13:45] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Donors reporting trouble with credit card fields using Firefox - https://phabricator.wikimedia.org/T404645#11280446 (10jgleeson) @KHancock99 this is fantastic! I'll see if we can figure out how to replicate it via the environment se... [12:57:02] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Donors reporting trouble with credit card fields using Firefox - https://phabricator.wikimedia.org/T404645#11281077 (10jgleeson) Thanks to that donor's report, I've tracked down the cause of the issue, at least for Firefox users. It... [13:46:28] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Donors reporting trouble with credit card fields using Firefox - https://phabricator.wikimedia.org/T404645#11281307 (10jgleeson) We've got it! {F66754678, size=full} [13:53:22] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Donors reporting trouble with credit card fields using Firefox - https://phabricator.wikimedia.org/T404645#11281332 (10krobinson) Nice job @jgleeson !!! And thank you @KHancock99 for the quick escalation / sharing of that incredib... [14:00:19] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 75.97, 78.32, 40.80 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:06:45] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Donors reporting trouble with credit card fields using Firefox - https://phabricator.wikimedia.org/T404645#11281370 (10jgleeson) Firefox donors can temporarily switch this off on a per-site basis, so in the **short term**, we could... [14:08:36] (03PS6) 10Ejegg: Delete a no-op function [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1190750 [14:09:01] (03CR) 10Ejegg: [C:03+2] "Re-applying wfan's C+2 to trigger jenkins" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1190750 (owner: 10Ejegg) [14:13:36] 06Fundraising-Backlog, 10FR-Donor-portal: Placeholder - Update magic link email from donor portal - https://phabricator.wikimedia.org/T407451#11281454 (10Ejegg) This can be done by non-fr-tech at https://civicrm.wikimedia.org/civicrm/admin/messageTemplates?reset=1#/edit?id=460 Note that it is also used to sen... [14:14:52] (03PS3) 10Ejegg: Donor Portal: link to email preferences page [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1195253 (https://phabricator.wikimedia.org/T407449) [14:15:22] 03Fundraising Sprint: Temp file pura, 06Fundraising-Backlog, 10FR-Donor-portal, 10FR-email-preferences, 13Patch-For-Review: Enable link to donors comms pref centre from donor portal - https://phabricator.wikimedia.org/T407449#11281469 (10Ejegg) [14:17:25] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1196698 [14:17:33] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1196698 (owner: 10Ejegg) [14:19:55] ACKNOWLEDGEMENT - check_kafkatee on frban2002 is CRITICAL: CRITICAL: kafka-jumbo1017:down, kafka-jumbo1008:init, kafka-jumbo1007:init, kafka-jumbo1009:init Jeff_Green known https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_kafkatee [14:20:17] RECOVERY - check_load on frban2002 is OK: LOAD OK - total load average: 2.06, 7.23, 21.63 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:31:43] (03Merged) 10jenkins-bot: Delete a no-op function [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1190750 (owner: 10Ejegg) [14:34:40] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07FR-Audience-Dev: Annual Recurring Statuses implementation in Civi (Audience Framework update) - https://phabricator.wikimedia.org/T377438#11281530 (10Lars) @MSuijkerbuijk_WMF I can confirm that all of the statuses and segments work in the same way: th... [14:35:20] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1196698 (owner: 10Ejegg) [14:39:30] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10FR-Donor-portal, 10FR-email-preferences: Add endowment tabs to donor portal contribution history - https://phabricator.wikimedia.org/T388756#11281554 (10Ejegg) Yep, this is done! Should we add a 'done' column on the donor portal board? Or just close it a... [14:51:57] !log donorwiki upgraded from d903982c to 70a7050f [14:51:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:02:21] 03Fundraising Sprint: Temp file pura, 06Fundraising-Backlog, 10FR-Donor-portal, 10FR-email-preferences, 13Patch-For-Review: Enable link to donors comms pref centre from donor portal - https://phabricator.wikimedia.org/T407449#11281621 (10Ejegg) a:03Ejegg [15:02:59] 03Fundraising Sprint: Temp file pura, 06Fundraising-Backlog, 07fr-current-sprint, 07payments-orchestration: Only download the scheduled gravy reports - https://phabricator.wikimedia.org/T396794#11281625 (10Ejegg) @cstone was there more to do on this? We've deployed the code and configuration. [15:28:11] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 49.82, 75.24, 44.57 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [15:30:15] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 115.08, 101.77, 59.97 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:33:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 50.55, 67.67, 50.16 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [15:35:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 56.33, 80.81, 63.28 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [15:51:12] (03PS1) 10Lars SG: Clarify donor segment and status descriptions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196712 (https://phabricator.wikimedia.org/T377438) [15:52:14] (03PS2) 10Lars SG: Clarify donor segment and status descriptions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196712 (https://phabricator.wikimedia.org/T377438) [15:53:11] RECOVERY - check_load on frban1002 is OK: LOAD OK - total load average: 3.00, 11.64, 24.21 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [16:00:15] RECOVERY - check_load on frban2002 is OK: LOAD OK - total load average: 2.78, 6.26, 23.13 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [16:05:09] 10fundraising-tech-ops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install franio1004 - https://phabricator.wikimedia.org/T405980#11281812 (10VRiley-WMF) [16:05:55] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07FR-Audience-Dev, 13Patch-For-Review: Annual Recurring Statuses implementation in Civi (Audience Framework update) - https://phabricator.wikimedia.org/T377438#11281818 (10Lars) @MSuijkerbuijk_WMF I have to correct my comment above (as I discovered th... [16:08:02] pcoombe: looks like PayPal is down globally [16:10:44] cstone beat me to it, thanks! [16:14:21] (03CR) 10Wfan: [C:03+2] Donor Portal: link to email preferences page [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1195253 (https://phabricator.wikimedia.org/T407449) (owner: 10Ejegg) [16:15:31] I'm logging off for the night, so if there's any updates on paypal please let emma and/or kelly know [16:15:39] 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:install (1) SSD each into franio200[1-3] - https://phabricator.wikimedia.org/T405982#11281893 (10Jhancock.wm) @Jgreen we got these drives in. I can install the one in franio2003 when i install the server. As for the other two, can i install the disks... [16:17:10] (03Merged) 10jenkins-bot: Donor Portal: link to email preferences page [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1195253 (https://phabricator.wikimedia.org/T407449) (owner: 10Ejegg) [16:18:08] (03CR) 10Wfan: "Should we add checksum check similar for emailPref for request new checksum if expired? We can still see the optin with wrong checksum, an" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1195274 (https://phabricator.wikimedia.org/T405047) (owner: 10Ejegg) [16:25:58] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: The ship is sinking! - https://phabricator.wikimedia.org/T407518 (10AKanji-WMF) 03NEW p:05Triage→03High [16:51:27] (03CR) 10Ejegg: "Sure, I'll add checksum expiration checking in a follow-on patch" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1195274 (https://phabricator.wikimedia.org/T405047) (owner: 10Ejegg) [16:51:49] (03PS8) 10Ejegg: Implement double opt-in page [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1195274 (https://phabricator.wikimedia.org/T405047) [16:53:20] (03PS6) 10Ejegg: Generic velocity filter [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1187920 (https://phabricator.wikimedia.org/T221378) [16:54:46] (03PS3) 10Lars SG: Clarify donor segment and status descriptions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196712 (https://phabricator.wikimedia.org/T377438) [16:54:49] (03CR) 10Ejegg: [C:03+2] Clarify donor segment and status descriptions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196712 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [17:11:49] (03Merged) 10jenkins-bot: Clarify donor segment and status descriptions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196712 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [17:25:36] 06Fundraising-Backlog, 10FR-Donor-portal: Placeholder - Interim Paypal recurring donors in Donor Portal - https://phabricator.wikimedia.org/T407527 (10AKanji-WMF) 03NEW [17:56:25] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07Epic: Send double opt-in emails to new contacts from RML - https://phabricator.wikimedia.org/T407534 (10Lars) 03NEW [17:59:02] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Send double opt-in confirmation message whenever a contact's email is changed - https://phabricator.wikimedia.org/T407261#11282413 (10Lars) @SHust asks specifically how we will handle Paypal email changes, referencing {T400765}. We will need to send a dou... [18:02:17] (03CR) 10Wfan: [C:03+2] "verified here https://wmf.localhost:32353/civicrm/admin/custom/group/fields#/?gid=26 and lgtm" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196477 (https://phabricator.wikimedia.org/T404240) (owner: 10Ejegg) [18:07:25] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Sync Preference Center Unsubscribes with Civi Fields - https://phabricator.wikimedia.org/T407536 (10SHust) 03NEW [18:10:57] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Add Snooze option and indicator for Preference Center - https://phabricator.wikimedia.org/T407539 (10SHust) 03NEW [18:14:02] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Preference Center Country Updates – Remove old country and clarify multi-country behavior - https://phabricator.wikimedia.org/T407540 (10SHust) 03NEW [18:18:18] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: The ship is sinking! - https://phabricator.wikimedia.org/T407518#11282516 (10jgleeson) 05Open→03Declined [18:20:02] (03Merged) 10jenkins-bot: Add activity_is_from_donor_portal custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196477 (https://phabricator.wikimedia.org/T404240) (owner: 10Ejegg) [18:26:45] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Secondary email receiving messages after merge - https://phabricator.wikimedia.org/T407541 (10SHust) 03NEW [18:39:16] (03PS2) 10Ejegg: Update existing email donations to have email as channel [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1194788 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [18:45:54] (03CR) 10Ejegg: "PS2: rebase + renumber upgrade script" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1194788 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [18:53:15] 03Fundraising Sprint: Temp file pura, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Edit Civi Grant Deadline reminder email message - https://phabricator.wikimedia.org/T150862#11282638 (10Lars) @CaitVirtue: @Eileenmcnaughton and I looked at a couple ways to try to make this happen as you propose, b... [18:57:01] 03Fundraising Sprint: Temp file pura, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Edit Civi Grant Deadline reminder email message - https://phabricator.wikimedia.org/T150862#11282652 (10Lars) @Eileenmcnaughton For the record, I looked at Scheduled Communications for this, but it just sends regula... [18:58:06] 03Fundraising Sprint: Temp file pura, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Edit Civi Grant Deadline reminder email message - https://phabricator.wikimedia.org/T150862#11282665 (10Lars) [19:12:32] (03PS1) 10Eileen: Fix money calc error [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196739 [19:13:55] (03CR) 10Ejegg: [C:03+2] Update existing email donations to have email as channel [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1194788 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [19:14:07] (03PS2) 10Eileen: Fix money calc error [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196739 [19:14:09] (03CR) 10Ejegg: [C:03+2] Fix money calc error [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196739 (owner: 10Eileen) [19:14:46] (03PS3) 10Eileen: Record chargebackreversals when received [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196563 (https://phabricator.wikimedia.org/T157456) [19:14:46] (03PS8) 10Eileen: Do not merge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1194132 [19:14:46] (03PS8) 10Eileen: Do not merge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1194154 [19:15:04] ok larssandergreen, just now getting to reviewing those tools patches [19:29:52] (03PS1) 10Eileen: Avoid type hint error [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1196742 [19:32:25] (03Merged) 10jenkins-bot: Update existing email donations to have email as channel [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1194788 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [19:32:26] (03Merged) 10jenkins-bot: Fix money calc error [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196739 (owner: 10Eileen) [19:33:47] (03CR) 10Ejegg: [C:03+2] Add both_funds_highest_native_currency to Acoustic export [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1196481 (https://phabricator.wikimedia.org/T407388) (owner: 10Lars SG) [19:35:02] (03Merged) 10jenkins-bot: Add both_funds_highest_native_currency to Acoustic export [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1196481 (https://phabricator.wikimedia.org/T407388) (owner: 10Lars SG) [19:35:26] (03PS2) 10Eileen: Add some additional options to limit the number of rows parsed [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195058 [19:36:50] (03PS1) 10Ejegg: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1196743 [19:36:56] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1196743 (owner: 10Ejegg) [19:37:47] (03Merged) 10jenkins-bot: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1196743 (owner: 10Ejegg) [19:42:05] (03CR) 10Ejegg: [C:04-1] "Think it needs a slightly different check for row limit" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195058 (owner: 10Eileen) [19:44:18] (03PS3) 10Eileen: Add some additional options to limit the number of rows parsed [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195058 [19:44:47] (03CR) 10Eileen: Add some additional options to limit the number of rows parsed (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195058 (owner: 10Eileen) [19:46:07] (03PS3) 10Eileen: Declare initial_scheme_transaction_id field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1193966 [19:46:09] (03CR) 10Ejegg: [C:03+2] Declare initial_scheme_transaction_id field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1193966 (owner: 10Eileen) [19:47:10] (03CR) 10Ejegg: [C:03+2] Avoid type hint error [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1196742 (owner: 10Eileen) [19:47:40] (03Merged) 10jenkins-bot: Avoid type hint error [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1196742 (owner: 10Eileen) [19:55:15] ok larssandergreen that's merged and ready to deploy [19:55:26] just looking at the stuff to run for deployment [19:56:26] (03PS1) 10Cstone: Update smash-pig to v0.9.11.6 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196745 [19:58:13] (03CR) 10Eileen: [C:03+2] Update smash-pig to v0.9.11.6 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196745 (owner: 10Cstone) [19:58:32] ejegg: if ^^ misses the deployment then I'll do another straight after [19:59:10] eileen: I haven't done any crm merges yet, just the tools repo [20:00:08] ok cool ejegg woud be good to merge that rowLimit one too - I fixed per your comment [20:00:14] oh cool [20:03:44] (03Merged) 10jenkins-bot: Declare initial_scheme_transaction_id field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1193966 (owner: 10Eileen) [20:06:55] (03CR) 10Ejegg: "oh oops, moving it up makes rowLimit=1 actually bail out before doing anything" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195058 (owner: 10Eileen) [20:09:22] ejegg: thanks. From what I can tell, it looks like we just need to run drop_incremental_schema and then rebuilkd_schema. Does that sound right to you? [20:17:55] (03Merged) 10jenkins-bot: Update smash-pig to v0.9.11.6 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196745 (owner: 10Cstone) [20:18:35] yep larssandergreen [20:19:29] !log fundraising python tools upgraded from 698309f1 to 3b0b3fc0 [20:19:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:20:20] ejegg: great, thanks for doing the deploy. Are you running those or should I? [20:20:33] I think I can [20:20:55] great, thanks [20:23:44] ok, i ran those scripts! [20:24:29] will just run the db updates (not the file builds) to check [20:25:02] (03PS4) 10Eileen: Add some additional options to limit the number of rows parsed [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195058 [20:25:08] ejegg: I updated it again ^^ [20:27:49] (03CR) 10Ejegg: [C:03+2] Add some additional options to limit the number of rows parsed [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195058 (owner: 10Eileen) [20:28:49] (03PS1) 10Eileen: Upgrade wikimedia/smash-pig to v0.9.11.6 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1196750 [20:29:01] (03CR) 10Eileen: [C:03+2] Upgrade wikimedia/smash-pig to v0.9.11.6 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1196750 (owner: 10Eileen) [20:29:33] hah my name in civi got all capitalized due to trustly [20:29:48] low confidence name source! [20:30:06] would we want to do that? [20:30:11] my address got updated by it too [20:30:22] guess thats fine to be capitalized [20:30:35] Or maybe we just fix casing of names from trustly? [20:31:52] eileen or ejegg: what is our intent for the acoustic export when a contact has a non-primary email that is also the primary email for another contact? do we think this should be suppressed or not? [20:32:11] it's exported with the other contact [20:32:16] at least, as far as I know [20:32:45] Would that be unwanted for some reason? [20:32:51] I guess duplicate records [20:33:02] ok, makes sense, that's what I observe - just wanted to check as some of the comments in the sql seem to suggest otherwise [20:34:07] in this particular edge case, the cid is for the contact with the non-primary, which is weird, but i think just a result of the order that we do things in [20:40:37] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1196751 [20:41:08] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1196751 (owner: 10Eileen) [20:41:33] (03PS4) 10Ejegg: Switch to hashes for error tracker [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1196522 [20:41:33] (03PS1) 10Ejegg: Put back removed fields from error summaries [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1196753 [20:45:51] (03Merged) 10jenkins-bot: Add some additional options to limit the number of rows parsed [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1195058 (owner: 10Eileen) [20:46:38] (03Merged) 10jenkins-bot: Upgrade wikimedia/smash-pig to v0.9.11.6 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1196750 (owner: 10Eileen) [20:46:38] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1196751 (owner: 10Eileen) [20:49:03] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1196755 [20:49:12] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1196755 (owner: 10Eileen) [20:50:08] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1196755 (owner: 10Eileen) [20:50:52] !log civicrm upgraded from ac4c185b to ab1d21dc [20:50:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:35:44] (03PS2) 10Cstone: Pass through the payment method so the recurring charge job can see it [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1195095 (https://phabricator.wikimedia.org/T375267) [21:48:48] (03PS1) 10Eileen: Fee amount as recorded on the contribution is the reverse [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196770 (https://phabricator.wikimedia.org/T152249) [21:53:21] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Secondary email receiving messages after merge - https://phabricator.wikimedia.org/T407541#11283485 (10Lars) @Shust In this case, the cause of the issue is that there was a duplicate contact (10567... [21:53:45] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Secondary email receiving messages after merge - https://phabricator.wikimedia.org/T407541#11283487 (10Lars) a:03Lars [21:58:53] (03CR) 10Lars SG: [C:03+2] Remove disabled financial types from local dev (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1192258 (https://phabricator.wikimedia.org/T405986) (owner: 10Eileen) [22:05:25] (03CR) 10CI reject: [V:04-1] Fee amount as recorded on the contribution is the reverse [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1196770 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [22:07:04] (03CR) 10Jgleeson: "I've smoke tested this and it does build the email the same. However, what do we expect to happen when the same ID/Key is written to? Will" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1196522 (owner: 10Ejegg) [22:14:59] (03Merged) 10jenkins-bot: Remove disabled financial types from local dev [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1192258 (https://phabricator.wikimedia.org/T405986) (owner: 10Eileen) [22:21:35] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: client side error for apple session exist already - https://phabricator.wikimedia.org/T407584 (10AnnWF) 03NEW [22:31:26] ejegg: i think we talked earlier about updating modified date for contacts to get them to update in the acoustic export (for the direct mail). Did you mean directly on the prod database? [22:31:31] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Donors reporting trouble with credit card fields using Firefox - https://phabricator.wikimedia.org/T404645#11283583 (10KHancock99) Great work @jgleeson. Thank you very much! Offering this donor big thanks and merch now ... [22:32:24] cccccbldjhujgcktuukchijbbrtirbdeknrdtgnvvgce [23:37:54] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 07payments-orchestration: Civi recurring donations display question - https://phabricator.wikimedia.org/T393227#11283629 (10Cstone) @AMJohnson thanks for all the feedback! 1. Yep they have been greyed out, I added in the missing options that the original pa... [23:53:04] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: client side error for apple session exist already - https://phabricator.wikimedia.org/T407584#11283670 (10AnnWF) Counts for month october: error-20251001.gz:85 error-20251002.gz:93 error-20251003.gz:180 error-20251004.gz:232 error-20251005.gz:209 error-20...