[00:01:00] 06Fundraising-Backlog: Add a cron job to update the local version currencyRate - https://phabricator.wikimedia.org/T409235 (10AnnWF) 03NEW [00:04:52] 10fundraising-tech-ops: Test task for automation - https://phabricator.wikimedia.org/T324628#11342847 (10Dwisehaupt) [00:15:21] (03CR) 10Wfan: [C:03+2] Remove obsolete contribution tracking fields [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1201786 (owner: 10Ejegg) [00:17:27] (03Merged) 10jenkins-bot: Remove obsolete contribution tracking fields [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1201786 (owner: 10Ejegg) [00:19:13] 06Fundraising-Backlog: Gravy transactions missing contribution_tracking_id - https://phabricator.wikimedia.org/T409237 (10Cstone) 03NEW [00:19:44] 06Fundraising-Backlog: Gravy transactions missing contribution_tracking_id - https://phabricator.wikimedia.org/T409237#11342899 (10Cstone) [00:24:47] (03PS22) 10Eileen: Add generateBatch api + test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1193485 (https://phabricator.wikimedia.org/T396071) [00:25:19] 06Fundraising-Backlog: Transactions missing contribution_tracking_id - https://phabricator.wikimedia.org/T409237#11342931 (10Cstone) [00:38:25] 10fundraising-tech-ops: Test task for automation - https://phabricator.wikimedia.org/T324628#11342963 (10Dwisehaupt) [00:42:05] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Automate - MGGO appeal update - https://phabricator.wikimedia.org/T406662#11342968 (10Eileenmcnaughton) @Lars @MDemosWMF Ideally for online gifts we do this on the way in - the difficulty is the li... [00:42:43] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing, 10FR-WMF-Audit: Automate - MGGO appeal update - https://phabricator.wikimedia.org/T406662#11342969 (10Eileenmcnaughton) [00:43:56] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Adyen, 10FR-Smashpig, 10FR-WMF-Audit: Deal with chargeback reversals - https://phabricator.wikimedia.org/T157456#11342975 (10Eileenmcnaughton) a:03Eileenmcnaughton [00:44:26] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Adyen, and 2 others: Deal with chargeback reversals - https://phabricator.wikimedia.org/T157456#11342978 (10Eileenmcnaughton) [00:44:39] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Adyen, and 2 others: Deal with chargeback reversals - https://phabricator.wikimedia.org/T157456#11342979 (10Eileenmcnaughton) [00:45:45] ejegg: since I just moved it to done - do you think 4 final story points is enough on https://phabricator.wikimedia.org/T157456 - it got a bit doggy [00:46:17] (03PS1) 10Eileen: Avoid double settlement [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201845 (https://phabricator.wikimedia.org/T152249) [01:06:34] (03PS1) 10Eileen: Add updates merged to upstream 6.8 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201848 [01:10:57] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 07Accessibility, 07fr-current-sprint: Post-Payment Monthly Convert accessibility - https://phabricator.wikimedia.org/T405941#11343043 (10KHancock99) Absolutely @AKanji-WMF , let me send you my presentation in Slack where I screen-recorded this issue and w... [01:15:11] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /srv 30789MiB (10% inode=97%): /dev 31930MiB (100% inode=99%): /run 6390MiB (99% inode=99%): / 3306MiB (47% inode=79%): /dev/shm 31960MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 290MiB (68% inode=99%): /home 11068MiB (89% inode=99%): /var 18883MiB (71% inode=82%): /tmp 12322MiB (99% inode=99%): /var/log 20398MiB (76% inode=99%): /var/tmp 422MiB (99 [01:15:11] 9%): /var/log/audit 422MiB (99% inode=99%): /run/user/12807 6392MiB (100% inode=99%): /run/user/3649 6392MiB (100% inode=99%): /run/user/33 6392MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [01:17:18] (03CR) 10Lars SG: [C:03+2] Add updates merged to upstream 6.8 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201848 (owner: 10Eileen) [01:17:21] (03Abandoned) 10Eileen: WIP but closer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1199771 (owner: 10Eileen) [01:19:32] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Clean up the fraud filter, to avoid block real donor - https://phabricator.wikimedia.org/T409245 (10AnnWF) 03NEW [01:20:09] RECOVERY - check_disk on civi1002 is OK: DISK OK - free space: /srv 48997MiB (16% inode=97%): /dev 31930MiB (100% inode=99%): /run 6390MiB (99% inode=99%): / 3306MiB (47% inode=79%): /dev/shm 31960MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 290MiB (68% inode=99%): /home 11068MiB (89% inode=99%): /var 18883MiB (71% inode=82%): /tmp 12322MiB (99% inode=99%): /var/log 20418MiB (76% inode=99%): /var/tmp 422MiB (99% inode=99% [01:20:09] og/audit 422MiB (99% inode=99%): /run/user/12807 6392MiB (100% inode=99%): /run/user/3649 6392MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [01:31:28] 06Fundraising-Backlog, 07Epic: Update exchange rate doc - https://phabricator.wikimedia.org/T409246 (10AnnWF) 03NEW [01:32:16] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 07Epic: Update exchange rate doc - https://phabricator.wikimedia.org/T409246#11343093 (10AnnWF) [01:32:38] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 07Epic: Update exchange rate doc - https://phabricator.wikimedia.org/T409246#11343095 (10AnnWF) https://phabricator.wikimedia.org/T409235 need update after this patch done [01:33:00] 06Fundraising-Backlog, 07Epic: Update exchange rate doc - https://phabricator.wikimedia.org/T409246#11343097 (10AnnWF) [01:34:09] (03Merged) 10jenkins-bot: Add updates merged to upstream 6.8 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201848 (owner: 10Eileen) [02:02:16] eileen: sure, I think 4 is appropriate - my half was just the sort of thing where I needed sustained concentration and was constantly being interrupted [02:03:24] cool - I was trying to recall what you were talking about for a bit there - it was a yucky one [02:10:19] (03PS7) 10Wfan: Show text to explain why active paypal ec not able to be modified [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1199532 (https://phabricator.wikimedia.org/T407527) [02:10:23] (03CR) 10Ejegg: [C:03+2] Show text to explain why active paypal ec not able to be modified [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1199532 (https://phabricator.wikimedia.org/T407527) (owner: 10Wfan) [02:15:19] (03PS2) 10Ejegg: Update exchange rates [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1201833 [02:19:46] (03PS1) 10Ejegg: Update CurrencyRates template [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201856 [02:23:01] eileen: yeah, I just was being a stickler for which field to mutate to indicate the reversal, and then it took me a while to get everything working once I tried to move that. [02:23:26] yeah - I had a lot of confusion even prior to that :-) [02:23:50] partly due to me thinking the ??= operator was null coalesce equality test and not assignment :P [02:24:19] though after it all I have a much better understanding of the rest of the changes you've been making towards the settlement batches! [02:25:02] I was concerned about getValidBatches always returning empty till I saw your comment about that logic not being finished [02:30:40] (03Merged) 10jenkins-bot: Show text to explain why active paypal ec not able to be modified [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1199532 (https://phabricator.wikimedia.org/T407527) (owner: 10Wfan) [02:35:25] (03PS23) 10Eileen: Add generateBatch api + test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1193485 (https://phabricator.wikimedia.org/T396071) [02:41:24] oh yeah getValidBatches - I need to work on making all the validation work as partof running jobs etc [02:43:33] (03CR) 10Cstone: [C:03+2] Avoid double settlement [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201845 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [02:58:20] (03CR) 10Eileen: [C:03+2] Update CurrencyRates template [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201856 (owner: 10Ejegg) [02:59:16] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1201859 [02:59:40] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1201859 (owner: 10Eileen) [03:00:26] (03Merged) 10jenkins-bot: Avoid double settlement [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201845 (https://phabricator.wikimedia.org/T152249) (owner: 10Eileen) [03:00:29] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1201859 (owner: 10Eileen) [03:01:11] !log civicrm upgraded from 3a637a8b to 090cd474 [03:01:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:16:39] (03CR) 10Cstone: [C:03+2] Add generateBatch api + test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1193485 (https://phabricator.wikimedia.org/T396071) (owner: 10Eileen) [03:21:35] !log config revision changed from 553c9c90 to 18e60944 [03:21:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:32:34] (03Merged) 10jenkins-bot: Add generateBatch api + test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1193485 (https://phabricator.wikimedia.org/T396071) (owner: 10Eileen) [03:51:09] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit, 07fr-current-sprint: Determine final exchange rate for Adyen settlements in non-USD - https://phabricator.wikimedia.org/T409249 (10Eileenmcnaughton) 03NEW [03:53:57] (03PS1) 10Lars SG: Update annual recurring segments and statuses [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201860 (https://phabricator.wikimedia.org/T377438) [03:54:55] (03CR) 10CI reject: [V:04-1] Update annual recurring segments and statuses [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201860 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [03:56:31] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201860 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [04:00:36] (03PS2) 10Lars SG: Update annual recurring segments and statuses [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201860 (https://phabricator.wikimedia.org/T377438) [04:07:00] (03CR) 10Lars SG: [C:03+2] Set is_major_gift on ingress and backfill prior data [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1193939 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [04:22:46] (03Merged) 10jenkins-bot: Set is_major_gift on ingress and backfill prior data [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1193939 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [04:24:14] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1201861 [04:24:28] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1201861 (owner: 10Eileen) [04:25:18] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1201861 (owner: 10Eileen) [04:31:15] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donor-portal: Change recurring donation frequency from monthly to yearly in the civi UI - https://phabricator.wikimedia.org/T396541#11343243 (10Lars) From @Eileenmcnaughton {F69902155} obv should be SmashPig processor... [04:38:25] !log civicrm upgraded from 090cd474 to 53b042e5 [04:38:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:51:38] (03PS1) 10Eileen: Fix ambiguous table name [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201862 (https://phabricator.wikimedia.org/T406193) [04:53:58] (03CR) 10Eileen: "OK - this works" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201860 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [04:59:34] (03CR) 10Eileen: [C:03+2] Add source/medium/campaign to more activities [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1200079 (https://phabricator.wikimedia.org/T407644) (owner: 10Ejegg) [05:00:00] (03PS1) 10Lars SG: Add annual recurring segments and statuses [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1201864 (https://phabricator.wikimedia.org/T407388) [05:02:53] (03CR) 10Lars SG: [C:03+2] Fix ambiguous table name [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201862 (https://phabricator.wikimedia.org/T406193) (owner: 10Eileen) [05:04:12] (03PS2) 10Lars SG: Add annual recurring segments and statuses [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1201864 (https://phabricator.wikimedia.org/T377438) [05:10:38] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1201865 [05:13:53] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1201865 (owner: 10Eileen) [05:14:43] !log civicrm upgraded from 53b042e5 to a7c697e9 [05:14:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:15:21] (03Merged) 10jenkins-bot: Add source/medium/campaign to more activities [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1200079 (https://phabricator.wikimedia.org/T407644) (owner: 10Ejegg) [05:56:31] (03PS4) 10Cstone: Send campaign, medium, source, and key to the donations queue [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1197740 (https://phabricator.wikimedia.org/T407179) [05:57:50] (03CR) 10CI reject: [V:04-1] Send campaign, medium, source, and key to the donations queue [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1197740 (https://phabricator.wikimedia.org/T407179) (owner: 10Cstone) [06:01:10] (03PS5) 10Cstone: Send campaign, medium, source, and key to the donations queue [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1197740 (https://phabricator.wikimedia.org/T407179) [06:09:02] (03PS1) 10Eileen: Fill account codes for new channels [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201867 (https://phabricator.wikimedia.org/T39607) [06:11:19] (03CR) 10Cstone: [C:03+2] Fill account codes for new channels [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201867 (https://phabricator.wikimedia.org/T39607) (owner: 10Eileen) [06:17:00] (03CR) 10Pppery: Fill account codes for new channels (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201867 (https://phabricator.wikimedia.org/T39607) (owner: 10Eileen) [06:23:13] (03PS2) 10Eileen: Fill account codes for new channels [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201867 (https://phabricator.wikimedia.org/T396071) [06:24:12] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201867 (https://phabricator.wikimedia.org/T396071) (owner: 10Eileen) [06:45:10] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1201871 [06:46:37] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1201871 (owner: 10Eileen) [06:47:47] !log civicrm upgraded from a7c697e9 to 1eeb1a46 [06:47:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:05:52] (03CR) 10Eileen: [C:03+2] Add annual recurring segments and statuses [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1201864 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [07:06:48] (03PS1) 10Eileen: Search kit for viewing batches [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201916 (https://phabricator.wikimedia.org/T399782) [07:07:17] (03Merged) 10jenkins-bot: Add annual recurring segments and statuses [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1201864 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [07:12:33] (03PS1) 10Eileen: Fix p to m [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201995 [07:13:07] (03CR) 10Eileen: [C:03+2] "new code - typo, self merge" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201995 (owner: 10Eileen) [07:29:57] (03Merged) 10jenkins-bot: Fix p to m [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201995 (owner: 10Eileen) [07:54:33] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1202035 [07:58:45] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1202035 (owner: 10Eileen) [07:59:41] !log ivicrm upgraded from 1eeb1a46 to 8efb2be1 [07:59:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:13:50] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 07fr-current-sprint: Map Settlement data to Civi field and table locations - https://phabricator.wikimedia.org/T397662#11343729 (10Eileenmcnaughton) @JMando this is the sql I'm using.... ` SELECT '10/30/2025' as DATE, CONCAT('Contribution Revenue... [13:56:05] (03CR) 10D3r1ck01: [C:03+1] Fix undefined keys on portal login page [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1198995 (owner: 10Ejegg) [14:37:12] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 96% used. Largest process: trino-server-co (16078) = 95.3% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [14:44:36] PROBLEM - check_load on fransw2002 is CRITICAL: LOAD CRITICAL - total load average: 119.51, 70.65, 40.04 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [14:44:36] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 99% used. Largest process: trino-server-co (16078) = 98.8% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [14:48:56] PROBLEM - check_load on fransw2002 is CRITICAL: LOAD CRITICAL - total load average: 82.41, 83.73, 53.51 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [14:49:00] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 99% used. Largest process: trino-server-co (16078) = 98.7% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [14:53:34] ACKNOWLEDGEMENT - check_load on fransw2002 is CRITICAL: LOAD CRITICAL - total load average: 82.41, 83.73, 53.51 Jeff_Green swapping. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [14:53:34] ACKNOWLEDGEMENT - check_memory on fransw2002 is CRITICAL: CRIT Memory 99% used. Largest process: trino-server-co (16078) = 98.7% Jeff_Green swapping. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [15:02:12] RECOVERY - check_load on fransw2002 is OK: LOAD OK - total load average: 0.66, 18.90, 38.72 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [15:02:14] RECOVERY - check_memory on fransw2002 is OK: OK Memory 1% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [15:38:03] (03PS1) 10Amire80: Remove comma splice from a message [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1202179 [15:39:24] (03CR) 10Ejegg: [C:03+2] "Thank you Amir, and all the best from fr-tech!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1202179 (owner: 10Amire80) [15:39:36] (03PS2) 10Thiemo Kreuz (WMDE): Remove comma splice from a message [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1202179 (owner: 10Amire80) [15:39:45] (03CR) 10Thiemo Kreuz (WMDE): [C:03+2] Remove comma splice from a message [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1202179 (owner: 10Amire80) [15:42:23] (03CR) 10Amire80: "❤️" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1202179 (owner: 10Amire80) [15:47:40] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: Downgrade pfw1-codfw to Junos 23.4R2-S3 - https://phabricator.wikimedia.org/T393996#11345299 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0cf41cdd-05b0-49da-ab43-1e9132f58a47) set by pt1979@cumin2002 for 2:00:00 on 1 host(... [16:19:34] (03Merged) 10jenkins-bot: Remove comma splice from a message [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1202179 (owner: 10Amire80) [16:34:00] PROBLEM - Host fransw2002 is DOWN: PING CRITICAL - Packet loss = 100% [16:36:55] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1202199 [16:36:58] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1202199 (owner: 10Ejegg) [16:37:12] RECOVERY - Host fransw2002 is UP: PING OK - Packet loss = 0%, RTA = 30.38 ms [16:43:36] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: Downgrade pfw1-codfw to Junos 23.4R2-S3 - https://phabricator.wikimedia.org/T393996#11345582 (10Papaul) 05Open→03Resolved Bother firewalls are not running Junos: 23.4R2-S5.5. Thanks to @Jgreen and @Dwisehaupt. Closing this task now [17:12:24] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 10FR-Donor-portal: Implement interim "change payment type" flow - https://phabricator.wikimedia.org/T396271#11345707 (10AKanji-WMF) This is good to go as per slack thread with donor portal group! cc: @AnnWF [17:28:16] (03CR) 10Lars SG: "I agree on the statuses. I think there is likely a lot of porosity between annual recurring and one-time giving in the real world. I think" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201860 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [17:39:02] (03PS1) 10Lars SG: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1202212 [17:40:10] (03CR) 10Lars SG: [C:03+2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1202212 (owner: 10Lars SG) [17:40:50] (03Merged) 10jenkins-bot: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1202212 (owner: 10Lars SG) [17:51:53] holy zuul backup batman [17:52:05] 1 hr 14 min in the queue for that merge to deployment [17:52:11] (03CR) 10Ejegg: [V:03+2 C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1202199 (owner: 10Ejegg) [17:53:07] !log payments-wiki upgraded from 8fe00530 to c2a4b377 [17:53:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:53:43] !log donorwiki upgraded from 8fe00530 to c2a4b377 [17:53:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:09:26] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 07Epic, 07fr-current-sprint, 07payments-orchestration: Gravy Braintree implementation: Venmo - https://phabricator.wikimedia.org/T375267#11345952 (10Cstone) Confirmed refunds are working as expected, and coming in on ipns [18:09:42] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 07Epic, 07fr-current-sprint, 07payments-orchestration: Gravy Braintree implementation: Venmo - https://phabricator.wikimedia.org/T375267#11345953 (10Cstone) [18:13:21] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 07Epic, 07fr-current-sprint, 07payments-orchestration: Gravy Braintree implementation: Venmo - https://phabricator.wikimedia.org/T375267#11345968 (10Cstone) confirmed the test transactions are in the audit files [18:13:30] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 07Epic, 07fr-current-sprint, 07payments-orchestration: Gravy Braintree implementation: Venmo - https://phabricator.wikimedia.org/T375267#11345969 (10Cstone) [18:14:12] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 07fr-current-sprint, 07payments-orchestration: Only download the scheduled gravy reports - https://phabricator.wikimedia.org/T396794#11345974 (10Cstone) Gravy is working on a fix for the consolidated file, it's currently stuck to my user account [18:23:01] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 07payments-orchestration: Review Adyen/Gravy webhook recipients list. - https://phabricator.wikimedia.org/T408590#11345981 (10Ejegg) Looks like we can disable the country-specific Adyen connections and their attendant webhooks for now - I guess they wer... [18:55:39] !log tools upgraded from 8e3ed11c to 773e8d11 [18:55:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:42:21] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit, and 2 others: CiviCRM / Analytics data clean up - https://phabricator.wikimedia.org/T406193#11346397 (10JMando) Just wanted to clarify, for "Major gifts is oK to be yes on subsequent - need to consider that... [19:42:45] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10FR-Donor-portal: Track Donor Portal navigation (with web beacons?) - https://phabricator.wikimedia.org/T404243#11346404 (10Dwisehaupt) @CHudson-WMF @AKanji-WMF @Ejegg We (fr-tech-ops) had some discussions about this today. With the beacon in place, we need... [19:51:14] (03PS1) 10Eileen: Append batch_reference to fee trxn_id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202260 (https://phabricator.wikimedia.org/T399782) [19:55:21] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donor-portal, and 2 others: Implement recurring donation monthly-to-annual conversion - https://phabricator.wikimedia.org/T388753#11346491 (10AnnWF) a:03AnnWF [20:04:01] (03CR) 10Cstone: [C:03+2] Append batch_reference to fee trxn_id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202260 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [20:13:17] (03CR) 10Eileen: [C:03+2] Update annual recurring segments and statuses [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201860 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [20:22:51] (03Merged) 10jenkins-bot: Append batch_reference to fee trxn_id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202260 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [20:33:42] (03Merged) 10jenkins-bot: Update annual recurring segments and statuses [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201860 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [20:45:18] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Coworker dedupe isn't running - https://phabricator.wikimedia.org/T408748#11346691 (10Lars) a:05Lars→03None [20:45:26] (03CR) 10Cstone: [C:03+2] "working locally, let's try it!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1191089 (https://phabricator.wikimedia.org/T356073) (owner: 10Ejegg) [20:47:32] thanks cstone ! [20:59:37] (03PS1) 10Eileen: Fix limit comparison [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202287 [21:00:15] (03PS1) 10Eileen: Fix fee trxn (part 2) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202288 (https://phabricator.wikimedia.org/T399782) [21:01:32] (03CR) 10Lars SG: [C:03+2] "There are some little details here that could be cleaned up, but I think good to merge and I can follow up with those instead of going bac" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201916 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [21:08:06] (03Merged) 10jenkins-bot: Send recurring failure email via coworker [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1191089 (https://phabricator.wikimedia.org/T356073) (owner: 10Ejegg) [21:13:50] (03PS1) 10Lars SG: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1202291 [21:16:02] (03CR) 10Lars SG: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1202291 (owner: 10Lars SG) [21:22:35] (03Merged) 10jenkins-bot: Search kit for viewing batches [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1201916 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [21:22:37] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1202291 (owner: 10Lars SG) [21:25:16] !log civicrm upgraded from 8efb2be1 to 40198c3f [21:25:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:32:34] (03PS2) 10Eileen: Fix fee trxn (part 2) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202288 (https://phabricator.wikimedia.org/T399782) [21:34:50] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit, 07fr-current-sprint: Determine final exchange rate for Adyen settlements in non-USD - https://phabricator.wikimedia.org/T409249#11346939 (10Eileenmcnaughton) Pending confirmation of amounts from Finance -... [21:35:29] 03Fundraising Sprint: Vixen, 06Fundraising-Backlog, 07Accessibility, 07fr-current-sprint: Post-Payment Monthly Convert accessibility - https://phabricator.wikimedia.org/T405941#11346941 (10AKanji-WMF) Thanks @KHancock99 - upon reviewing the video it's clear how the screen reader ignores the modal: {F69917173} [21:38:22] 06Fundraising-Backlog, 10FR-ActiveMQ, 10FR-WMF-Audit, 07Epic: [Epic] Do error log parsing via queue - https://phabricator.wikimedia.org/T133413#11346956 (10Eileenmcnaughton) 05Open→03Declined Framing is a bit dated here [21:43:23] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit, 07payments-orchestration: New data flows in support of reconciliation related to the Gravy payment orchestration project - https://phabricator.wikimedia.org/T366840#11346982 (10Eileenmcnaughton) @EMartin a... [21:44:23] 03Fundraising Sprint: Vixen, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Channel QA - https://phabricator.wikimedia.org/T407179#11346990 (10Eileenmcnaughton) [21:46:41] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Get Paypal audit processing Intactable - https://phabricator.wikimedia.org/T409362 (10Eileenmcnaughton) 03NEW [21:49:29] 03Fundraising Sprint: Vixen, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Channel QA - https://phabricator.wikimedia.org/T407179#11347019 (10Eileenmcnaughton) a:03Eileenmcnaughton [21:58:59] (03PS1) 10Lars SG: Update annual recurring segments and statuses [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1202303 (https://phabricator.wikimedia.org/T377438) [22:00:52] eileen: could you look at the trigger updates above? I've looked them over and they look right (left side top of the file are duplicated below and many of the changes are removing end of line spaces for some reason) [22:02:40] oh if you saved the file using your IDE it will have trimmed those [22:03:49] (03CR) 10Eileen: [C:03+2] Update annual recurring segments and statuses [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1202303 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [22:05:17] (03Merged) 10jenkins-bot: Update annual recurring segments and statuses [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1202303 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [22:14:44] (03PS1) 10Lars SG: Update annual recurring segments and statuses triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202307 (https://phabricator.wikimedia.org/T377438) [22:18:23] (03PS2) 10Lars SG: Update annual recurring segments and statuses triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202307 (https://phabricator.wikimedia.org/T377438) [22:19:42] eileen: ok, it turns out I did that on deployment instead of master anyways, so i'll revert and above is one against master, without removing the end of line spaces [22:20:31] (03CR) 10Eileen: [C:03+2] Update annual recurring segments and statuses triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202307 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [22:20:41] Ah I've done that before... [22:24:57] (03CR) 10Cstone: [C:03+2] Fix fee trxn (part 2) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202288 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [22:30:33] (03PS1) 10Lars SG: Revert Update annual recurring segments and statuses [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1202317 (https://phabricator.wikimedia.org/T377438) [22:37:56] (03PS1) 10Wfan: Tie venmo username to use the one without @ for all Since it's not stable pass over to civi [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202319 (https://phabricator.wikimedia.org/T406153) [22:42:38] (03Merged) 10jenkins-bot: Update annual recurring segments and statuses triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202307 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [22:45:55] (03Merged) 10jenkins-bot: Fix fee trxn (part 2) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202288 (https://phabricator.wikimedia.org/T399782) (owner: 10Eileen) [22:47:07] 06Fundraising-Backlog: Transactions missing contribution_tracking_id - https://phabricator.wikimedia.org/T409237#11347274 (10Eileenmcnaughton) [22:54:53] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Odd dlocal recurrings - https://phabricator.wikimedia.org/T409369 (10Eileenmcnaughton) 03NEW [22:58:04] (03PS2) 10Wfan: Tie venmo username to use prefix @ for all [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202319 (https://phabricator.wikimedia.org/T406153) [22:58:25] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Odd dlocal recurrings - https://phabricator.wikimedia.org/T409369#11347301 (10Eileenmcnaughton) [23:00:02] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Odd dlocal recurrings - https://phabricator.wikimedia.org/T409369#11347312 (10Eileenmcnaughton) [23:03:48] eileen: can you +2 this one reverting deployment: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1202317 [23:04:08] (03CR) 10Eileen: [C:03+2] Revert Update annual recurring segments and statuses [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1202317 (https://phabricator.wikimedia.org/T377438) (owner: 10Lars SG) [23:15:25] (03CR) 10CI reject: [V:04-1] Tie venmo username to use prefix @ for all [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202319 (https://phabricator.wikimedia.org/T406153) (owner: 10Wfan) [23:20:15] (03PS3) 10Wfan: Tie venmo username to use prefix @ for all [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202319 (https://phabricator.wikimedia.org/T406153) [23:22:13] (03PS1) 10Lars SG: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1202327 [23:26:35] (03CR) 10Lars SG: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1202327 (owner: 10Lars SG) [23:27:31] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1202327 (owner: 10Lars SG) [23:35:23] !log civicrm upgraded from 40198c3f to 0f49dd1d [23:35:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:38:20] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Odd dlocal recurrings - https://phabricator.wikimedia.org/T409369#11347416 (10Eileenmcnaughton) @Ejegg just thought I'd check this one with you - they might be fine - in which case we can just merge the first contact & c... [23:39:15] (03CR) 10CI reject: [V:04-1] Tie venmo username to use prefix @ for all [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202319 (https://phabricator.wikimedia.org/T406153) (owner: 10Wfan) [23:40:33] (03PS4) 10Wfan: Tie venmo username to use prefix @ for all [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202319 (https://phabricator.wikimedia.org/T406153) [23:43:43] (03PS2) 10Eileen: Fix limit comparison [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202287 (https://phabricator.wikimedia.org/T383194) [23:47:35] (03CR) 10Wfan: [C:03+2] Fix limit comparison [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1202287 (https://phabricator.wikimedia.org/T383194) (owner: 10Eileen)