[00:45:34] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: URGENT: Merge Behavior Changed — Causing RML Emails to Be Sent in Error! - https://phabricator.wikimedia.org/T401752#11084389 (10Eileenmcnaughton) I ran a one off query ` DELETE s FROM silverpop_export_staging s INNER JOIN civic... [00:55:28] (03PS2) 10Wfan: Add is contribution has paypal in epc to determin if show email edit in epc [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178090 (https://phabricator.wikimedia.org/T401006) [01:10:01] (03PS1) 10Wfan: only show edit email from epc if donor never made donation with paypal [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178658 (https://phabricator.wikimedia.org/T401006) [01:16:21] (03CR) 10CI reject: [V:04-1] Add is contribution has paypal in epc to determin if show email edit in epc [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178090 (https://phabricator.wikimedia.org/T401006) (owner: 10Wfan) [01:19:42] (03PS1) 10Ejegg: Clean up dynamic property in Adyen tests [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178659 [01:20:09] (03CR) 10CI reject: [V:04-1] Clean up dynamic property in Adyen tests [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178659 (owner: 10Ejegg) [01:29:36] (03PS1) 10Ejegg: Fix dynamic properties in Braintree tests [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178661 [01:30:08] (03CR) 10CI reject: [V:04-1] Fix dynamic properties in Braintree tests [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178661 (owner: 10Ejegg) [01:32:16] (03PS1) 10Ejegg: Fix substr on null in AmazonMessage [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178662 [01:33:04] (03CR) 10CI reject: [V:04-1] Fix substr on null in AmazonMessage [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178662 (owner: 10Ejegg) [01:34:41] (03Abandoned) 10Eileen: Handle unconfigured redis session password [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177513 (owner: 10Eileen) [01:37:51] (03PS2) 10Ejegg: Clean up dynamic property in Adyen tests [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178659 [01:37:51] (03PS2) 10Ejegg: Fix dynamic properties in Braintree tests [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178661 [01:37:51] (03PS2) 10Ejegg: Fix substr on null in AmazonMessage [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178662 [01:54:04] hmm, if I let fetchMessageByGatewayOrderId fetch resolved data I need to make a lot more changes than I had hoped [01:55:18] To properly test that they're marked resolved I'm trying to add the is_resolved property to the returned array, but that makes a lot of other tests unhappy [01:55:53] and would mean another thing to unset before merging to the donation queue message (right now the only thing that's NOT compatible with that format is the pending_id) [01:56:29] I could add another method argument to allow fetching resolved messages, but that seems a bit smelly [02:02:56] (03PS5) 10Ejegg: WIP update PendingDatabase for is_resolved [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1174076 (https://phabricator.wikimedia.org/T400304) [02:02:57] (03PS1) 10Ejegg: WIP add is_resolved property to pending messages [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178672 [02:03:13] there's the attempt to add the is_resolved property ^^^ [02:03:40] (03CR) 10CI reject: [V:04-1] WIP update PendingDatabase for is_resolved [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1174076 (https://phabricator.wikimedia.org/T400304) (owner: 10Ejegg) [02:03:46] (03CR) 10CI reject: [V:04-1] WIP add is_resolved property to pending messages [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178672 (owner: 10Ejegg) [02:03:54] so maybe instead, I just want to query the db directly in the 8 failed tests from the first patch [02:04:03] and check for the is_resolved property there [02:36:19] (03PS3) 10Wfan: Add is contribution has paypal in epc to determin if show email edit in epc [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178090 (https://phabricator.wikimedia.org/T401006) [04:15:56] (03PS4) 10Eileen: CiviCRM 6.6 master (do not merge) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1175975 [04:15:56] (03PS2) 10Eileen: Update composer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177559 [04:15:57] (03PS2) 10Eileen: Add currency to tests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177574 [04:15:57] (03PS2) 10Eileen: Adjust for upstream format change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177575 [04:19:05] 03Fundraising Sprint: Pirate's Booty, 06Fundraising-Backlog, 10fundraising-tech-ops, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Review Redis session handler locking - https://phabricator.wikimedia.org/T400848#11084550 (10Eileenmcnaughton) @Jgreen not quite ... It turns out the password needs... [04:29:22] (03PS4) 10Wfan: instead of update primary email via EPC add email as emailPreference [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1164497 (https://phabricator.wikimedia.org/T396523) [04:31:56] (03CR) 10CI reject: [V:04-1] Adjust for upstream format change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177575 (owner: 10Eileen) [04:32:03] (03CR) 10CI reject: [V:04-1] CiviCRM 6.6 master (do not merge) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1175975 (owner: 10Eileen) [04:32:24] (03Abandoned) 10Eileen: Try a different place for the patch [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1176331 (owner: 10Eileen) [04:33:06] (03CR) 10CI reject: [V:04-1] Update composer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177559 (owner: 10Eileen) [04:37:42] (03CR) 10CI reject: [V:04-1] Add currency to tests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177574 (owner: 10Eileen) [04:43:29] (03PS1) 10Wfan: WIP: design needed, send verification email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178692 (https://phabricator.wikimedia.org/T396523) [05:31:42] (03PS5) 10Eileen: CiviCRM 6.6 master (do not merge) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1175975 [05:31:42] (03PS3) 10Eileen: Update composer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177559 [05:31:42] (03PS3) 10Eileen: Add currency to tests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177574 [05:31:42] (03PS3) 10Eileen: Adjust for upstream format change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177575 [05:31:43] (03PS1) 10Eileen: Do not set address_primary.id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178710 [05:35:41] (03PS1) 10Eileen: Update test to reflect upstream class change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178711 [05:46:42] (03CR) 10CI reject: [V:04-1] Add currency to tests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177574 (owner: 10Eileen) [05:47:51] (03CR) 10CI reject: [V:04-1] Adjust for upstream format change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177575 (owner: 10Eileen) [05:48:35] (03CR) 10CI reject: [V:04-1] Update composer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177559 (owner: 10Eileen) [05:49:02] (03CR) 10CI reject: [V:04-1] CiviCRM 6.6 master (do not merge) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1175975 (owner: 10Eileen) [05:50:14] (03CR) 10CI reject: [V:04-1] Do not set address_primary.id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178710 (owner: 10Eileen) [05:51:56] (03CR) 10CI reject: [V:04-1] Update test to reflect upstream class change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178711 (owner: 10Eileen) [06:34:31] (03PS1) 10Eileen: Do exchange rate conversion in Import hook [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178715 [06:48:24] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06Release-Engineering-Team, 10Continuous-Integration-Config: CI build - missing patch & mysqli? - https://phabricator.wikimedia.org/T401451#11084631 (10Eileenmcnaughton) 05Resolved→03Open @hashar @dancy -sorry to re-open I just discovered there... [06:51:13] (03CR) 10CI reject: [V:04-1] Do exchange rate conversion in Import hook [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178715 (owner: 10Eileen) [06:52:35] eileen: I thought CiviCRM CI got fixed? :) [06:53:02] hashar: it's the composer job still needing mysql [06:54:04] ie the top one is failing for 'us reasons' but composer works - but the bottom one is missing mysqli https://usercontent.irccloud-cdn.com/file/3Fn1JDXe/image.png [06:54:19] mysqli not mysql [06:54:25] AHH [06:54:30] cause CiviCRM now requires that extension [06:54:49] makes you wonder why the PHP community went with multiple extensions.. [06:55:12] well mysql is basically defunct in favour of mysqli since about php 8 [06:55:56] What changed is we started using composer-merge-plugin to combine CiviCRM requirements with our wmf ones & hence we got mysqli added [06:56:19] (before that we just hoped they didn't conflict :-) [06:56:53] hope is the sin of engineers. That always ends backfiring us in the end [06:57:00] anyway, I'll add php-mysqli [06:59:36] thanks !!! [07:00:02] but let's cling to hope cos I still have quite a lot of hope-driven code [07:09:21] so I went confused cause we never install `php-mysqli` [07:09:42] but that is because the debian package is named `php-mysql` while the PHP extension is mysqli [07:14:23] which means the patch I have made to add php-mysqli was needless [07:14:27] that is an alias to php-mysql [07:14:30] anyway [07:19:35] eileen: I think I will add composer to the releng/civicrm image [07:19:50] and change the composer test job to use the releng/civicrm image which already has the mysqli php extension [07:19:59] and this way the linter will use the same env as the test image [07:20:05] hashar: thanks that sounds great [07:29:55] (03PS1) 10Eileen: Update test mapping to Contribution.note [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178720 [07:41:34] I have updated the job to at least have the composer image installed [07:41:35] I'll test it [07:42:35] (03PS1) 10Hashar: (DO NOT SUBMIT) test with "composer" in the CI image [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178724 [07:42:49] that will take a while, I gotta cleanup/refactor some other images in the process [07:46:09] cool [07:47:02] (03CR) 10CI reject: [V:04-1] Update test mapping to Contribution.note [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178720 (owner: 10Eileen) [08:08:33] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178751 (owner: 10L10n-bot) [08:18:25] (03PS4) 10Eileen: Adjust for upstream format change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177575 [08:18:25] (03PS2) 10Eileen: Do not set address_primary.id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178710 [08:18:26] (03PS2) 10Eileen: Update test to reflect upstream class change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178711 [08:18:26] (03PS2) 10Eileen: Do exchange rate conversion in Import hook [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178715 [08:18:27] (03PS2) 10Eileen: Update test mapping to Contribution.note [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178720 [08:37:44] (03CR) 10CI reject: [V:04-1] Update test to reflect upstream class change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178711 (owner: 10Eileen) [08:38:01] (03CR) 10CI reject: [V:04-1] Update test mapping to Contribution.note [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178720 (owner: 10Eileen) [08:38:07] (03CR) 10CI reject: [V:04-1] Adjust for upstream format change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177575 (owner: 10Eileen) [08:38:54] (03CR) 10CI reject: [V:04-1] Do not set address_primary.id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178710 (owner: 10Eileen) [08:39:54] (03CR) 10CI reject: [V:04-1] Do exchange rate conversion in Import hook [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178715 (owner: 10Eileen) [08:40:05] (03Abandoned) 10Hashar: (DO NOT SUBMIT) test with "composer" in the CI image [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178724 (owner: 10Hashar) [08:50:13] I guess I will summarize on the task [08:50:28] but that took a bit longer than expected cause I had to refactor things [08:51:12] I will also make it so that `composer test` is run before the CiviCRM buildkit tests: https://gerrit.wikimedia.org/r/c/integration/config/+/1178814/1/zuul/layout.yaml [08:51:31] so that if there is a linting error, CI reports immediately and does not bother running the slow tests [08:53:09] (03PS5) 10Eileen: Adjust for upstream format change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177575 [08:53:09] (03PS3) 10Eileen: Do not set address_primary.id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178710 [08:53:09] (03PS3) 10Eileen: Update test to reflect upstream class change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178711 [08:53:09] (03PS3) 10Eileen: Do exchange rate conversion in Import hook [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178715 [08:53:10] (03PS3) 10Eileen: Update test mapping to Contribution.note [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178720 [09:11:04] (03CR) 10CI reject: [V:04-1] Adjust for upstream format change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177575 (owner: 10Eileen) [09:12:23] (03CR) 10CI reject: [V:04-1] Update test to reflect upstream class change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178711 (owner: 10Eileen) [09:12:55] (03CR) 10CI reject: [V:04-1] Do exchange rate conversion in Import hook [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178715 (owner: 10Eileen) [09:13:37] (03CR) 10CI reject: [V:04-1] Update test mapping to Contribution.note [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178720 (owner: 10Eileen) [09:13:47] (03CR) 10CI reject: [V:04-1] Do not set address_primary.id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178710 (owner: 10Eileen) [09:24:52] (03Restored) 10Hashar: (DO NOT SUBMIT) test with "composer" in the CI image [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178724 (owner: 10Hashar) [09:25:10] (03CR) 10Hashar: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178724 (owner: 10Hashar) [09:26:36] (03CR) 10Hashar: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177559 (owner: 10Eileen) [09:34:11] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06Release-Engineering-Team, 10Continuous-Integration-Config, 13Patch-For-Review: CI build - missing patch & mysqli? - https://phabricator.wikimedia.org/T401451#11085098 (10hashar) I had to do a bit of refactoring. I have changed the composer test j... [09:38:44] (03CR) 10CI reject: [V:04-1] (DO NOT SUBMIT) test with "composer" in the CI image [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178724 (owner: 10Hashar) [10:17:19] (03CR) 10Hashar: "recheck after https://gerrit.wikimedia.org/r/c/integration/config/+/1178826 I forgot to update the composer job to `releng/civicrm:0.9` wh" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177559 (owner: 10Eileen) [10:19:26] (03Abandoned) 10Hashar: (DO NOT SUBMIT) test with "composer" in the CI image [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178724 (owner: 10Hashar) [10:19:40] All done I think, :] [10:29:16] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06Release-Engineering-Team, 10Continuous-Integration-Config, 13Patch-For-Review: CI build - missing patch & mysqli? - https://phabricator.wikimedia.org/T401451#11085519 (10hashar) I have removed the `composer-php82` job which uses an image that does... [10:37:54] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: Move pfw1b-codfw to rack F5 - https://phabricator.wikimedia.org/T401297#11085585 (10ayounsi) From a quick look it does seem best to have two control links for proper redundancy. But I suggest that we do a test. In a maintenance window, unplug t... [12:19:29] (03PS1) 10Damilare Adedoyin: Remove validation fields for venmo payments [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178854 (https://phabricator.wikimedia.org/T400413) [13:24:53] (03Abandoned) 10Raimond Spekking: Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178429 (owner: 10L10n-bot) [13:26:10] (03CR) 10Raimond Spekking: [C:03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178751 (owner: 10L10n-bot) [13:32:12] PROBLEM - check_load on fransw2002 is CRITICAL: LOAD CRITICAL - total load average: 120.80, 58.90, 31.84 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [13:34:12] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: Move pfw1b-codfw to rack F5 - https://phabricator.wikimedia.org/T401297#11086240 (10Papaul) @ayounsi I will have to work with fundraising to see when it will be best for us to do so. Thanks [13:37:12] RECOVERY - check_load on fransw2002 is OK: LOAD OK - total load average: 1.82, 25.67, 25.30 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_load [13:58:38] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: URGENT: Merge Behavior Changed — Causing RML Emails to Be Sent in Error! - https://phabricator.wikimedia.org/T401752#11086362 (10jgleeson) @Eileenmcnaughton, I can confirm that the affected donor is now on the suppression list. @S... [14:01:43] 03Fundraising Sprint: Pirate's Booty, 06Fundraising-Backlog, 10fundraising-tech-ops, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Review Redis session handler locking - https://phabricator.wikimedia.org/T400848#11086370 (10Jgreen) >>! In T400848#11084550, @Eileenmcnaughton wrote: > @Jgreen not qu... [14:05:15] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM: DAF batch data entry - match to Fidelity rules - https://phabricator.wikimedia.org/T392723#11086384 (10MDemosWMF) This can be closed out since it will be included in the new batch data entry tool. [14:06:11] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM: Searchkit build - Donors from Engage import who need TYLs - https://phabricator.wikimedia.org/T396068#11086389 (10MDemosWMF) I have figured this one out, so it can be closed out! [14:15:52] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10FR-email-preferences: adding email template for email confirmation - https://phabricator.wikimedia.org/T396521#11086421 (10AnnWF) {F65749884} {F65751022} @RLopez-WMF thanks, I do not have approved content to use, this is the draft email and verification... [14:33:52] PROBLEM - check_load on fransw2003 is CRITICAL: LOAD CRITICAL - total load average: 117.76, 82.82, 52.37 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_load [14:33:52] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 99% used. Largest process: trino-server-co (1813545) = 98.6% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [14:33:54] PROBLEM - check_procs on fransw2003 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_procs [14:36:20] PROBLEM - check_load on fransw2003 is CRITICAL: LOAD CRITICAL - total load average: 61.97, 80.60, 58.81 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_load [14:36:22] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 100% used. Largest process: trino-server-co (1813545) = 98.4% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [14:36:22] RECOVERY - check_procs on fransw2003 is OK: PROCS OK: 624 processes https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_procs [14:37:12] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 95% used. Largest process: trino-server-co (1098055) = 93.8% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [14:41:14] RECOVERY - check_load on fransw2003 is OK: LOAD OK - total load average: 0.43, 29.64, 42.63 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_load [14:41:14] RECOVERY - check_memory on fransw2003 is OK: OK Memory 0% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [14:41:20] PROBLEM - check_trino on fransw2003 is CRITICAL: connect to address 127.0.0.1 and port 8443: Connection refused https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_trino [14:42:14] RECOVERY - check_memory on fransw2002 is OK: OK Memory 43% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [14:45:06] ACKNOWLEDGEMENT - check_trino on fransw2003 is CRITICAL: connect to address 127.0.0.1 and port 8443: Connection refused Jeff_Green investigating https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_trino [14:46:18] RECOVERY - check_trino on fransw2003 is OK: HTTP OK: HTTP/1.1 200 OK - 834 bytes in 0.104 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_trino [14:49:27] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: URGENT: Merge Behavior Changed — Causing RML Emails to Be Sent in Error! - https://phabricator.wikimedia.org/T401752#11086543 (10greg) p:05Unbreak!→03High (changing from UBN to High as this should be resolved, we just wait to ac... [14:58:25] (03PS14) 10Damilare Adedoyin: Create donor portal page for recurring donation pause process. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1176526 (https://phabricator.wikimedia.org/T388749) [15:07:18] PROBLEM - check_trino on fransw2002 is CRITICAL: connect to address 127.0.0.1 and port 8443: Connection refused https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_trino [15:12:18] PROBLEM - check_trino on fransw2002 is CRITICAL: connect to address 127.0.0.1 and port 8443: Connection refused https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_trino [15:13:52] (03PS14) 10Damilare Adedoyin: WIP: Create donor portal page for recurring donation cancel process. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178053 (https://phabricator.wikimedia.org/T388749) [15:17:18] RECOVERY - check_trino on fransw2002 is OK: HTTP OK: HTTP/1.1 200 OK - 836 bytes in 0.114 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_trino [15:18:49] 03Fundraising Sprint: Pirate's Booty, 06Fundraising-Backlog, 10fundraising-tech-ops, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Review Redis session handler locking - https://phabricator.wikimedia.org/T400848#11086641 (10Jgreen) @eileen I think this is fixed, although I'm not sure what output I... [15:21:14] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 96% used. Largest process: trino-server-co (2457773) = 95.8% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [15:26:14] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 96% used. Largest process: trino-server-co (2457773) = 95.8% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [15:26:25] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10FR-email-preferences: adding email template for email confirmation - https://phabricator.wikimedia.org/T396521#11086689 (10SHust) @AnnWF Great call, and I agree! I’d suggest making it prominent so donors definitely don’t miss it. [15:27:32] ACKNOWLEDGEMENT - check_memory on fransw2003 is CRITICAL: CRIT Memory 96% used. Largest process: trino-server-co (2457773) = 95.8% Jeff_Green known https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [15:41:14] RECOVERY - check_memory on fransw2003 is OK: OK Memory 0% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [15:41:20] PROBLEM - check_trino on fransw2003 is CRITICAL: connect to address 127.0.0.1 and port 8443: Connection refused https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_trino [15:46:20] RECOVERY - check_trino on fransw2003 is OK: HTTP OK: HTTP/1.1 200 OK - 832 bytes in 0.108 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_trino [15:48:32] fr-tech I have some pretty trivial patches to clean up deprecation warnings on smashig tests: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/SmashPig/+/1178662 (and its two parents) [15:48:43] looking ejegg [15:52:36] (03CR) 10Jgleeson: [C:03+2] Clean up dynamic property in Adyen tests [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178659 (owner: 10Ejegg) [15:52:48] (03CR) 10Jgleeson: [C:03+2] Fix dynamic properties in Braintree tests [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178661 (owner: 10Ejegg) [15:53:31] (03Merged) 10jenkins-bot: Clean up dynamic property in Adyen tests [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178659 (owner: 10Ejegg) [15:53:31] (03Merged) 10jenkins-bot: Fix dynamic properties in Braintree tests [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178661 (owner: 10Ejegg) [15:56:45] thanks jgleeson ! [15:57:19] I was just looking at that last substr fix to confirm we're happy sending back '' if it's not set and I think that's fine [15:57:29] I'm just debating what to do about that is_resolved flag. that is, whether to add it to the message that's returned [15:57:44] oh would substr return null on a null input? [15:57:50] (03CR) 10Jgleeson: [C:03+2] Fix substr on null in AmazonMessage [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178662 (owner: 10Ejegg) [15:57:53] I thought it would always return a string even then [15:58:27] (03Merged) 10jenkins-bot: Fix substr on null in AmazonMessage [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178662 (owner: 10Ejegg) [15:59:29] I think it returned false ejegg actually so not really that useful [15:59:33] ooh weird [15:59:41] '' / null is better [15:59:51] so for the patch immediately on top of those fixes: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/SmashPig/+/1174076 [16:00:00] I'm currently leaning towards restoring PS3 [16:00:03] which had passing tests [16:01:28] and made the fetch function implicitly filter by is_resolved=0 [16:01:37] Changelog: 8.0.0 The function returns an empty string where it previously returned false. [16:01:46] oho [16:01:52] looks like it is an empty string [16:01:54] anyway [16:01:57] lgtm [16:02:07] so I think PS3 is the least risky way to start persisting pending messages [16:02:45] so we'll have some data to use for audit resolution (replacing log_hunt_and_send) [16:03:41] anyway, I guess we'd want to add another function to do the lookup without that filter [16:03:57] (or maybe another fn arg, but that's a bit code-smelly) [16:12:22] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10FR-email-preferences: adding email template for email confirmation - https://phabricator.wikimedia.org/T396521#11086890 (10AnnWF) @RLopez-WMF any suggestion? We have have this part merge first, since we already hide the edit email for paypal donor, maybe... [16:16:03] i swear i had seen a way in gerrit to restore a previous PS [16:16:15] i can always just d/l it and re-review [16:17:15] hmm [16:17:48] you could download it with `git-review -d $PS/3` [16:17:59] yah, am restoring it that way for now [16:18:08] and then git ammend and push [16:18:10] ah yeah cool [16:18:13] I wanted to update the commit message anyway [16:20:06] (03PS6) 10Ejegg: Update PendingDatabase for is_resolved [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1174076 (https://phabricator.wikimedia.org/T400304) [16:20:18] OK, I think ^^^ should be the safest way forward [16:20:37] oh, let [16:20:48] s also do something about that deleteOldMessages [16:21:08] I guess there we really do want both a deleteOld and resolveOld function [16:21:42] since we want to mark things resolved as soon as we can't do anything about them in the pending transaction resolver (i.e. the paypal express checkout session ends) [16:21:52] and then actually delete them after a few months [16:34:41] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Uruguay recurring erroring while using Gr4vy link - https://phabricator.wikimedia.org/T394406#11086981 (10jgleeson) a:03jgleeson [16:34:42] (03PS1) 10Wfan: WIP email verification [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178912 (https://phabricator.wikimedia.org/T396523) [16:37:33] (03CR) 10CI reject: [V:04-1] WIP email verification [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178912 (https://phabricator.wikimedia.org/T396523) (owner: 10Wfan) [16:38:25] (03PS1) 10Wfan: Add hint text for paypal donor [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178913 (https://phabricator.wikimedia.org/T401006) [16:42:08] (03CR) 10CI reject: [V:04-1] Add hint text for paypal donor [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178913 (https://phabricator.wikimedia.org/T401006) (owner: 10Wfan) [16:52:36] (03PS2) 10Wfan: Add hint text for paypal donor [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178913 (https://phabricator.wikimedia.org/T401006) [16:58:01] (03PS3) 10Wfan: Add hint text for paypal donor [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178913 (https://phabricator.wikimedia.org/T401006) [17:06:14] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3961 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:11:14] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2916 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:16:14] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2075 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:26:14] RECOVERY - check_mysql on frdb2005 is OK: Uptime: 1117547 Threads: 4 Questions: 93308879 Slow queries: 596 Opens: 1649 Open tables: 1292 Queries per second avg: 83.494 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:34:02] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: URGENT: Merge Behavior Changed — Causing RML Emails to Be Sent in Error! - https://phabricator.wikimedia.org/T401752#11087166 (10SHust) @Eileenmcnaughton, will the affected donors' secondary email be opted out? I checked a few of th... [18:01:41] (03PS15) 10Damilare Adedoyin: Create donor portal page for recurring donation cancel process. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178053 (https://phabricator.wikimedia.org/T388749) [18:03:57] (03CR) 10Wfan: [C:03+2] Remove mw head from DonorPortal template [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1177988 (owner: 10Damilare Adedoyin) [18:06:28] (03Merged) 10jenkins-bot: Remove mw head from DonorPortal template [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1177988 (owner: 10Damilare Adedoyin) [18:13:34] 03Fundraising Sprint: Pirate's Booty, 06Fundraising-Backlog, 07fr-current-sprint, 10MW-1.45-notes (1.45.0-wmf.13; 2025-08-05), 13Patch-For-Review: Do not show email update for donor ever donated with paypal - https://phabricator.wikimedia.org/T401006#11087329 (10AnnWF) show edit if not donate with paypal... [18:30:45] (03PS2) 10Wfan: WIP email verification [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178912 (https://phabricator.wikimedia.org/T396523) [18:46:21] (03CR) 10Ejegg: "Looks pretty good - I think there are a couple of things that a follow-on patch can clean up." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174561 (owner: 10Eileen) [18:50:28] hmm, failmails... [18:51:46] ah, deadlocks, but I guess they've cleared up [18:55:09] looks like somebody's doing some deduping, but the queries mostly seem to be well behaved [18:59:24] (03CR) 10Jgleeson: [C:03+2] "LGTM!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1174076 (https://phabricator.wikimedia.org/T400304) (owner: 10Ejegg) [18:59:40] thanks jgleeson ! [18:59:43] NP [18:59:45] -caps [18:59:53] (03Merged) 10jenkins-bot: Update PendingDatabase for is_resolved [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1174076 (https://phabricator.wikimedia.org/T400304) (owner: 10Ejegg) [19:00:41] OK, let me tag a version and shut off queues for deploy - I think I'll want to make sure that goes out to standalone + CRM vendor dir at the same time. [19:02:36] (03PS1) 10Ejegg: WIP Maint script to mark old pendings resolved [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178936 (https://phabricator.wikimedia.org/T400304) [19:03:16] (03Abandoned) 10Ejegg: WIP add is_resolved property to pending messages [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178672 (owner: 10Ejegg) [19:04:03] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1178938 [19:04:05] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1178938 (owner: 10Ejegg) [19:04:42] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1178938 (owner: 10Ejegg) [19:05:17] (03PS8) 10Eileen: Add audit action and allow it to pass out to the settle action. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174561 [19:05:25] (03CR) 10Ejegg: [C:03+2] Add audit action and allow it to pass out to the settle action. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174561 (owner: 10Eileen) [19:13:46] (03PS2) 10Wfan: WIP: design needed, send verification email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178692 (https://phabricator.wikimedia.org/T396523) [19:15:55] (03PS1) 10Ejegg: Update SmashPig to 0.9.7.8 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178940 [19:15:57] (03CR) 10Ejegg: [C:03+2] Update SmashPig to 0.9.7.8 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178940 (owner: 10Ejegg) [19:17:13] (03CR) 10Jgleeson: "Thanks Dami. It looks good but I wanted to confirm we're intentionally dropping the checks on email, first_name & last_name, along with th" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178614 (https://phabricator.wikimedia.org/T400413) (owner: 10Damilare Adedoyin) [19:17:13] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10FR-email-preferences: adding email template for email confirmation - https://phabricator.wikimedia.org/T396521#11087545 (10RLopez-WMF) Hi @AnnWF thanks for sharing the screenshot. I can work on a template draft while we get final content approval [19:26:33] (03Merged) 10jenkins-bot: Add audit action and allow it to pass out to the settle action. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174561 (owner: 10Eileen) [19:26:34] oh i might as well update the vendor dir in payments too, to get that ACH email fix out [19:27:21] (03PS1) 10Ejegg: Update SmashPig to 0.9.7.8 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178943 [19:27:23] (03CR) 10Ejegg: [C:03+2] Update SmashPig to 0.9.7.8 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178943 (owner: 10Ejegg) [19:29:14] we are having db deadlock in civi1002, and I realized that we no longer use drush, ejegg do you know how to go db from civi1002? [19:37:20] (03Merged) 10jenkins-bot: Update SmashPig to 0.9.7.8 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178940 (owner: 10Ejegg) [19:39:28] (03PS2) 10Ejegg: Maintenance script to mark old pendings resolved [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178936 (https://phabricator.wikimedia.org/T400304) [19:39:50] (03PS1) 10Ejegg: Avoid deprecation warning passing null to trim() [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178946 [19:40:21] wfan: we can use innotop and the query killer script [19:40:30] I think it's in failmailzoo [19:40:56] https://wikitech.wikimedia.org/wiki/Fundraising/Internal-facing/CiviCRM/Debugging_deadlocks it's not giving me any connect id [19:41:03] https://wikitech.wikimedia.org/wiki/Fundraising/Data_and_flow/Failmail_zoo [19:41:23] oh [19:41:24] lemme try [19:41:34] Ah this one with Q works [19:42:08] Thanks all~ [19:42:14] I had seen a couple dedupe queries with user 199, but nothing that stuck around for very long [19:43:03] I killed it [19:43:18] it was an fr_stats query running for 5 minutes [19:43:19] yeah I saw you killed it, thanks [19:43:31] we might need to let analytics folks know [19:43:40] I'll ping them on Slack [19:43:55] Thanks Jack~ [19:48:10] (03Merged) 10jenkins-bot: Update SmashPig to 0.9.7.8 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178943 (owner: 10Ejegg) [19:49:12] (03PS3) 10Wfan: WIP email verification [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178912 (https://phabricator.wikimedia.org/T396523) [19:52:46] (03PS2) 10Damilare Adedoyin: Remove validation for venmo payments [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178614 (https://phabricator.wikimedia.org/T400413) [19:53:29] fr-tech: should the analytics folks be pulling from prod civicrm during busy periods? I thought we had a slave db running on frdev with the same data? [19:54:26] hmm, yeah, i was pretty sure their queries were set up not to interfere with writes [19:54:54] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: make wmf-cv sql:cli works - https://phabricator.wikimedia.org/T398378#11087632 (10jgleeson) [19:58:17] (03CR) 10Damilare Adedoyin: "Thanks for the CR jgleeson, like with paypal we want to keep the field requirements consistent with the direct connection. Currently, the " [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178614 (https://phabricator.wikimedia.org/T400413) (owner: 10Damilare Adedoyin) [19:59:18] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1178951 [19:59:21] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1178951 (owner: 10Ejegg) [20:00:45] damilare|away: thanks for clarifying on removing the field checks. That other comment was to highlight that it says paypal in the venmo validator comment [20:01:11] Sure I've made the change on that jgleeson, thanks for spotting [20:02:02] oh you have, sorry when I saw the responses on gerrit it still showed the old one. I didn't realise it doesn't show you latest! my bad [20:02:06] thanks for the updates [20:02:20] no worries, thanks for the CR! [20:03:47] sure thing [20:09:21] (03PS1) 10Ejegg: Update SmashPig [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1178953 [20:09:30] (03CR) 10Ejegg: [C:03+2] Update SmashPig [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1178953 (owner: 10Ejegg) [20:10:12] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1178954 [20:10:14] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1178954 (owner: 10Ejegg) [20:16:46] I'm going to turn off the queue jobs for deployment [20:16:54] well, I guess it's a bit early [20:24:18] !log stopped scheduled fundraising jobs for deployment [20:24:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:24:37] (03Merged) 10jenkins-bot: Update SmashPig [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1178953 (owner: 10Ejegg) [20:24:37] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1178954 (owner: 10Ejegg) [20:27:09] !log fundraising civicrm upgraded from 321a17c0 to 5949c641 [20:27:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:27:58] !log standalone SmashPig upgraded from 83293ee1 to 3d1d6f15 [20:27:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:31:38] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1178951 (owner: 10Ejegg) [20:32:40] (03PS1) 10Ejegg: Update composer.lock for new SmashPig [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1178956 [20:32:43] (03CR) 10Ejegg: [C:03+2] Update composer.lock for new SmashPig [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1178956 (owner: 10Ejegg) [20:37:19] !log payments-wiki upgraded from cd876775 to 5c5b4637 [20:37:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:43:16] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 5352 5000, payments-init is 2241 2000, pending is 2532 2000 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 20 keys, up 79 days 4 hours - memory use is 272.38M (peak 296.50M, 3.47% of max, fragmentation 1.04%), connected_slaves is 3, donations is 1255, jobs is 0, jobs-adyen is 30, jobs-dlocal is 0, jobs-paypal is 748, recurring is 43, refund is [20:43:16] scribe is 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [20:45:00] (03Merged) 10jenkins-bot: Update composer.lock for new SmashPig [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1178956 (owner: 10Ejegg) [20:48:14] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 6729 5000, payments-init is 2826 2000, pending is 3515 2000 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 20 keys, up 79 days 4 hours - memory use is 276.17M (peak 296.50M, 3.51% of max, fragmentation 1.04%), connected_slaves is 3, donations is 1601, jobs is 0, jobs-adyen is 45, jobs-dlocal is 0, jobs-paypal is 951, recurring is 48, refund is [20:48:14] scribe is 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [20:53:16] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 8173 5000, payments-init is 3429 2000, pending is 4518 2000 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 21 keys, up 79 days 4 hours - memory use is 280.16M (peak 296.50M, 3.56% of max, fragmentation 1.04%), connected_slaves is 3, donations is 1997, jobs is 0, jobs-adyen is 53, jobs-dlocal is 0, jobs-paypal is 1149, recurring is 54, refund is [20:53:16] bscribe is 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [20:58:18] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 9491 5000, payments-init is 4004 2000, pending is 5439 2000 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 21 keys, up 79 days 4 hours - memory use is 283.94M (peak 296.50M, 3.60% of max, fragmentation 1.04%), connected_slaves is 3, donations is 2344, jobs is 0, jobs-adyen is 61, jobs-dlocal is 0, jobs-paypal is 1312, recurring is 57, refund is [20:58:18] bscribe is 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [20:58:52] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fundraiser Public Data Export job is causing deadlocks - https://phabricator.wikimedia.org/T401960 (10jgleeson) 03NEW [20:58:59] ACKNOWLEDGEMENT - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 9491 5000, payments-init is 4004 2000, pending is 5439 2000 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 21 keys, up 79 days 4 hours - memory use is 283.94M (peak 296.50M, 3.60% of max, fragmentation 1.04%), connected_slaves is 3, donations is 2344, jobs is 0, jobs-adyen is 61, jobs-dlocal is 0, jobs-paypal is 1312, recurring is 57, r [20:58:59] 32, unsubscribe is 0 Jeff_Green yup. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [21:09:19] ok, let me turn the jobs back on to stop those queue complaints [21:12:53] !log re-enabled fundraising scheduled jobs [21:12:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:18:02] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fundraiser Public Data Export job is causing deadlocks - https://phabricator.wikimedia.org/T401960#11087883 (10jgleeson) [21:20:55] (03CR) 10Jgleeson: [C:03+2] "Thanks for confirming!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178614 (https://phabricator.wikimedia.org/T400413) (owner: 10Damilare Adedoyin) [21:21:25] (03Merged) 10jenkins-bot: Remove validation for venmo payments [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178614 (https://phabricator.wikimedia.org/T400413) (owner: 10Damilare Adedoyin) [21:21:47] thanks for the CR jgleeson [21:21:55] np! [21:23:17] RECOVERY - check_redis on frqueue1003 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 16 keys, up 79 days 5 hours - memory use is 261.48M (peak 296.50M, 3.34% of max, fragmentation 1.05%), connected_slaves is 3, donations is 3620, jobs is 0, jobs-adyen is 0, jobs-dlocal is 0, jobs-paypal is 3, payments-antifraud is 4, payments-init is 2, pending is 0, recurring is 4, refund is 5, unsubscribe is 0 https://icinga.wikim [21:23:17] cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [21:28:50] (03PS1) 10Damilare Adedoyin: Small fixes to tests to reuse mock data objects and remove unused Footer component [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178967 [21:32:43] 06Fundraising-Backlog, 06FR-donorrelations: Email address field on donation form suggesting yahoo.ca - https://phabricator.wikimedia.org/T401738#11087957 (10MBeat33) 1711641 is another ticket, from a US donor. [21:43:57] 06Fundraising-Backlog: process_unsubscribes log activity failed due to contact_id not exit - https://phabricator.wikimedia.org/T401967 (10AnnWF) 03NEW [22:08:03] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fundraiser Public Data Export job is causing deadlocks - https://phabricator.wikimedia.org/T401960#11088030 (10greg) We discussed this internally briefly and I agree that we should turn off the frdata update job until we can determine the root cause and m... [22:21:01] 06Fundraising-Backlog, 06FR-donorrelations: Email address field on donation form suggesting yahoo.ca - https://phabricator.wikimedia.org/T401738#11088068 (10Ejegg) Sorry, we've just updated the mailcheck library. I'll see if I can make the new version stop doing that. [22:43:25] (03PS1) 10Wfan: add existance check before doing unsubscribe action [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178980 (https://phabricator.wikimedia.org/T401967) [22:46:11] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fundraiser Public Data Export job is causing deadlocks - https://phabricator.wikimedia.org/T401960#11088115 (10greg) `lang=irc 15:16 22:16:44 frpm1002: jgleeson ssh from frbast1002 15:19 22:19:55 frpm1002: fundraising... [22:46:20] (03PS1) 10Ejegg: Add yahoo.com back to list of mailcheck domains [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178981 (https://phabricator.wikimedia.org/T401738) [23:01:59] (03CR) 10CI reject: [V:04-1] add existance check before doing unsubscribe action [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178980 (https://phabricator.wikimedia.org/T401967) (owner: 10Wfan) [23:35:01] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: process_unsubscribes log activity failed due to contact_id not exit - https://phabricator.wikimedia.org/T401967#11088157 (10AnnWF) a:03AnnWF