[00:56:19] (03CR) 10AndyRussG: [V: 03+2 C: 03+2] "Fantastic, thanks so much for this!! :) :)" [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/710574 (owner: 10Jgleeson) [01:34:40] (03CR) 10AndyRussG: [V: 03+2 C: 03+2] "Fantastic, thanks so much!!!! :)" [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/712783 (owner: 10Ejegg) [01:49:09] (03CR) 10AndyRussG: [V: 03+2 C: 03+2] "Ahhh thanks and apologies this was missed!! :)" [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/712755 (owner: 10Ejegg) [03:23:25] (03PS11) 10AndyRussG: Add ApplePay payments-wiki override setup options [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/711216 (https://phabricator.wikimedia.org/T288772) (owner: 10Jgleeson) [06:19:42] (03CR) 10jerkins-bot: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/713146 (owner: 10L10n-bot) [06:24:25] (03CR) 10Raimond Spekking: [C: 03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/713146 (owner: 10L10n-bot) [07:07:51] 10Fundraising-Backlog, 10fundraising Sprint NULL calorie food cart, 10fundraising sprint onion pit, 10FR-France, 10MW-1.37-notes (1.37.0-wmf.19; 2021-08-16): Adding tax disambiguation clause to Payments Wiki in France - https://phabricator.wikimedia.org/T286880 (10Damilare) Great @CDenes_WMF , this was e... [15:25:31] ooh, class ContributionTrackingDataValidationException not found [15:27:38] oh, probably the Civi autoloader isn't working in that queue consumer [15:28:26] so... we want to init civi for it, I guess [15:28:46] Hi fr-tech [15:28:53] hi cstone! [15:29:02] and hi fr-tech [15:32:58] Hi ejegg [15:35:40] Hi ejegg [15:35:46] Hi cstone [15:36:10] Hi damilare ! [16:04:47] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: server error seen when attempting to export long search kit list - https://phabricator.wikimedia.org/T287995 (10DStrine) [16:04:56] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Fredge report only lets you export <200 rows - https://phabricator.wikimedia.org/T287994 (10DStrine) [16:05:14] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising Sprint NULL calorie food cart: Test relationship giving history with search kit - https://phabricator.wikimedia.org/T286742 (10DStrine) [16:09:32] (03PS1) 10Ejegg: WIP move contribution_tracking QC into Civi [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/713290 [16:17:28] dang, those qc failures are still happening [16:20:18] (03CR) 10jerkins-bot: [V: 04-1] WIP move contribution_tracking QC into Civi [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/713290 (owner: 10Ejegg) [16:24:05] is it a bad message ejegg ? [16:26:33] looks like it must be cstone [16:26:40] that's what would put it down the codepath [16:26:50] 10Fundraising-Backlog, 10FR-Adyen, 10FR-Japan: Japan mm/dd/yy field order - https://phabricator.wikimedia.org/T288849 (10DStrine) [16:26:53] shoot, let me at least put up a temporary patch that just inits CiviCRM [16:27:03] before finishing that bigger migration patch [16:28:18] 10Fundraising-Backlog, 10FR-Adyen, 10FR-Japan: Japan mm/dd/yy field order - https://phabricator.wikimedia.org/T288849 (10DStrine) Thanks @EMartin one issue we have is that we can't change the expiration date formatting based on country. That would be the easiest fix for us but would be a request to the PSPs.... [16:30:23] (03PS1) 10Ejegg: Quick fix for contribution tracking queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/713294 [16:30:37] cstone: I think ^^^ should be enough to at least get it loading that class [16:30:50] let's see, maybe I can mess with a message locally to trigger that [16:31:05] right, just delete the id [16:31:08] ah okay I can try too [16:34:16] oh huh, it finds the class now but still doesn't send it to the damaged table [16:34:27] i'd like to at least see what these messages are [16:34:35] Warning, throwing an unknown exception: "25" [16:35:02] I think it wasn't necessary to add a new constant for CONTRIBUTION_TRACKING over in WmfException [16:41:15] bah, can't even insert into the damaged table without a gateway value? [16:41:19] well that's no fun [16:41:52] ok, let's at least make sure the bad message is logged [16:42:54] ok, seems to be logged like 4 times [16:43:18] hah [16:43:43] (03PS2) 10Ejegg: Quick fix for contribution tracking queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/713294 [16:43:47] 10Fundraising-Backlog, 10fundraising-tech-ops: Only retain fredge data for 2 years - https://phabricator.wikimedia.org/T288971 (10DStrine) [16:44:17] so the 'fatal' => TRUE should make sure we still get failmail on bad c_t messages [16:45:44] ooh, those contact import refactor patches are in master and not deployed [16:45:56] think I'll put that quick fix up as a cherry-pick for starters [16:46:05] we should really get at the root of this [16:46:06] I can [16:46:31] 't think of a user-controlled way to get the id blanked [16:46:38] in messages sent to that queue [16:47:03] oh, let's see if we have some good messages still coming in [16:47:20] yeah the id blank is weird [16:51:19] 10Fundraising-Backlog, 10fundraising-tech-ops: Remove old/no longer used tables in pgehres - https://phabricator.wikimedia.org/T288973 (10DStrine) [16:54:47] most messages look good ejegg [16:55:00] yep, that's reassuring [16:55:54] i guess figure out what happened at 4am EST that looks like when it started [16:56:30] just waiting on zuuls hard work then I'll +2 that ejegg [16:56:54] thanks cstone [16:58:25] (03CR) 10Cstone: [C: 03+2] "Looks good" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/713294 (owner: 10Ejegg) [17:03:56] (03PS1) 10Ejegg: Quick fix for contribution tracking queue [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/713296 [17:04:07] 10Wikimedia-Fundraising-Banners, 10donate.wikimedia.org: Don't ask to "pay the fee" if it would take donation over maximum amount - https://phabricator.wikimedia.org/T228540 (10Pcoombe) I've kind of done this on donatewiki, as we have a new major gifts amount set with 12000 USD as the max ask. So we obviously... [17:04:12] ^^ that's the cherry-pick [17:04:28] (03CR) 10Ejegg: [C: 03+2] Quick fix for contribution tracking queue [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/713296 (owner: 10Ejegg) [17:09:00] (03Merged) 10jenkins-bot: Quick fix for contribution tracking queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/713294 (owner: 10Ejegg) [17:09:27] (03CR) 10Ejegg: [V: 03+2 C: 03+2] Quick fix for contribution tracking queue [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/713296 (owner: 10Ejegg) [17:11:03] !log updated fundraising CiviCRM from f3895dc907 to 1e32084622 [17:11:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:38:01] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising sprint onion pit: Search kit links not loading when sharing out to Engage - https://phabricator.wikimedia.org/T279686 (10MDemosWMF) Hi @Eileenmcnaughton just walked through this with Engage and they **do** have access and data is loading f... [18:14:29] oh derp, cstone, the raw messages were totally in the logs [18:15:40] ok, so they were from 'CiviCRM' / direct / run27930 [18:15:57] so is that the recurring charge job? [18:16:11] looking at another contribution from that same run [18:17:55] hmm, nope, it's a paypal_ec one with no contribution_recur_id [18:18:55] odd, and the donation itself seems to have been sent from payments-wiki [18:20:04] wait, i'm confusing myself [18:20:19] that db dive was for a contribution from the same run [18:20:49] which DID have contribution tracking [18:20:56] now let's look at the one that didn't [18:21:46] ok, the one that didn't have a ct_id is from the listener [18:22:35] so it's the donations queue consumer which tried to send a ct.contribution_id update even though it didn't have the ct.id [18:22:58] i started looking to see if the donations queue consumer fail from around the same time was related but got distracted [18:24:14] ok, so the 'custom' field is empty on the IPN for that one [18:57:49] 10Fundraising-Backlog, 10fundraising-tech-ops, 10fr-donorservices: Civi certificate request: Lefki - https://phabricator.wikimedia.org/T288986 (10MBeat33) [18:58:46] 10Fundraising-Backlog, 10fundraising-tech-ops, 10fr-donorservices: Civi certificate request: Yamini - https://phabricator.wikimedia.org/T288987 (10MBeat33) [19:00:45] 10Fundraising-Backlog, 10fundraising-tech-ops, 10fr-donorservices: Civi certificate request: Yamini - https://phabricator.wikimedia.org/T288987 (10MBeat33) re: Civi certificate approval request for Lefki and Yamini Megan Hernandez 2:38 PM (21 minutes ago) to me, Jessica, Kristie Yes, I can approve while... [19:00:57] 10Fundraising-Backlog, 10fundraising-tech-ops, 10fr-donorservices: Civi certificate request: Lefki - https://phabricator.wikimedia.org/T288986 (10MBeat33) re: Civi certificate approval request for Lefki and Yamini Megan Hernandez 2:38 PM (21 minutes ago) to me, Jessica, Kristie Yes, I can approve while... [19:09:45] ejegg: we got another ct failmail the one you added is nice for not having to dig [19:10:46] heh, at least it's more convenient [19:11:32] ok, another paypal_ec one [19:12:59] ahh, i think our guard against these is just a lil weak [19:13:10] should be a check against empty [19:15:19] (03PS1) 10Ejegg: Better guard against empty c_t.id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/713310 [19:15:35] cstone: I think that patch should fix it [19:16:21] ah makes sense [19:23:41] ah damn, what is this getPreferredLanguage() weirdness [19:24:09] I was seeing that on my local when I tried upgrading to the new RC before disabling the data translate extension [19:24:26] ah no, not that exactly [19:24:56] ahh, so the message just had nothing for language [19:25:12] hmm, again from the paypal IPN listener [19:26:37] ahh, and yep you might be right about the failmail having to do with that contribution tracking id issue [19:26:46] ejegg: what do you expect to happen locally with a missing id in the ct message? im getting an email error but i think that might just be some local setup thing [19:27:13] oh, yeah, i would expect the qc to stop and send a failmail [19:27:21] ok cool [19:27:23] maybe failmail isn't configured in docker? [19:27:36] yeah the error is a email config thing [19:54:51] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops: Add user = 1 to drush alias - https://phabricator.wikimedia.org/T288708 (10DStrine) [19:57:43] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2021, 10Patch-For-Review: Swap all our process logging to Civi::log('wmf') from watchdog() - https://phabricator.wikimedia.org/T288585 (10DStrine) [20:14:05] 10Fundraising-Backlog: Catch Adyen Checkout API errors on the DI side - https://phabricator.wikimedia.org/T288830 (10XenoRyet) [20:18:24] 10Fundraising-Backlog, 10fr-matching-gifts: Add process-control job to run the drush civicrm-matching-gifts-employers-check command - https://phabricator.wikimedia.org/T260726 (10XenoRyet) [20:21:22] 10Fundraising-Backlog, 10fr-matching-gifts: Test out the end-to-end matching gifts file deployment process. - https://phabricator.wikimedia.org/T273558 (10XenoRyet) [21:07:35] 10Fundraising-Backlog: Dlocal SSL Certificate due to expire 20 Aug 2021 - https://phabricator.wikimedia.org/T287386 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt We are not pinning this certificate. Nothing to do here. [21:08:58] 10fundraising-tech-ops: Update dns_to_ipset to not use domain name as key for set - https://phabricator.wikimedia.org/T287416 (10Dwisehaupt) 05Open→03Resolved ` [frack::puppet::private] fbea8c3 Remove duplicate ipset rulenames civicrm, payments, payments_listener ` This is complete. Closing. [21:09:11] 10fundraising-tech-ops: Update dns_to_ipset to not use domain name as key for set - https://phabricator.wikimedia.org/T287416 (10Dwisehaupt) [21:13:51] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/713326 [21:15:35] ejegg: all good to deploy https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/713326 - includes the save change + your exception change [21:16:13] ? [21:20:06] yep eileen [21:20:14] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/713326 (owner: 10Eileen) [21:20:17] actually I've got one more lil patch for the save [21:20:29] oh ok - i should wait for that? [21:20:41] (03PS1) 10Ejegg: Ensure numeric value passed for language lookup [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/713328 [21:22:00] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops: Add user = 1 to drush alias - https://phabricator.wikimedia.org/T288708 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt This is complete and should be pushed by puppet any minute now. ` [frack::puppet] 06484ddc Specify use... [21:22:20] just that little one eileen https://gerrit.wikimedia.org/r/713328 [21:22:44] oh and this too if you're up for it: https://gerrit.wikimedia.org/r/713310 [21:23:08] ejegg: that's fine - but elsewhere I've used (int) to cast to int so intval is less consistent [21:23:48] oh ok, i'll swap it [21:25:35] (03CR) 10Eileen: [C: 03+2] Better guard against empty c_t.id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/713310 (owner: 10Ejegg) [21:30:12] (03PS2) 10Ejegg: Ensure numeric value passed for language lookup [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/713328 [21:30:16] ok, swapped the cast ^^ [21:31:03] (03CR) 10Eileen: [C: 03+2] Ensure numeric value passed for language lookup [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/713328 (owner: 10Ejegg) [21:33:54] (03CR) 10Eileen: [C: 03+2] "recheck" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/713326 (owner: 10Eileen) [21:36:18] (03Merged) 10jenkins-bot: Better guard against empty c_t.id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/713310 (owner: 10Ejegg) [21:39:49] ejegg: those 2 have delayed it to the point where I can't deploy before being AFK for an hour - so I'll either hold off or you can deploy once yours are both merged [21:40:07] oh shoot, i'll deploy mine later then! [21:41:48] (03Merged) 10jenkins-bot: Ensure numeric value passed for language lookup [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/713328 (owner: 10Ejegg) [21:43:10] ejegg: yeah - I mean I can deploy - but I just won't be around for an hour if anything weird happens - & I wouldn't normally deploy & go offline [21:43:50] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/713332 [21:43:58] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/713332 (owner: 10Eileen) [21:44:33] yours are up now anyway - I have to shoot away for an hour like I said (high finance trading - 20 empty egg carrtons for one full one) [21:44:42] but deploy if you want or I will when I'm back [21:44:53] lol [21:45:09] ok, I'll hold off [21:59:04] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising Sprint NULL calorie food cart, 10fundraising sprint onion pit: Create new WMF donor fields - https://phabricator.wikimedia.org/T288721 (10EYener) Adding @JMando to keep track of changes. @Eileenmcnaughton just so I understand it: the cur... [22:47:07] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: server error seen when attempting to export long search kit list - https://phabricator.wikimedia.org/T287995 (10Dwisehaupt) We are using the standard nginx and apache settings for these. The relevant sections of documentation are: Nginx: large_client_hea... [23:10:28] ok - back - I'll deploy [23:10:57] cool! [23:11:16] shall we turn off the donations QC out of an abundance of caution? [23:14:53] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: server error seen when attempting to export long search kit list - https://phabricator.wikimedia.org/T287995 (10Eileenmcnaughton) @Dwisehaupt how do you feel about increasing it? I am not 100% sure how long it would need to be to get the string in that th... [23:15:42] !log civicrm revision changed from 1e32084622 to ba0c7705bb, config revision is 7bdc78073d [23:15:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:18:40] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: server error seen when attempting to export long search kit list - https://phabricator.wikimedia.org/T287995 (10Eileenmcnaughton) I guess if we look at ^^ then 8000 rows would be 8000 bytes * the length of the id - which could be up to 8 digits (I don't t... [23:32:38] 10Fundraising Sprint Ketchup Flume Ride, 10Fundraising Sprint Mandatory corn dogs, 10Fundraising-Backlog, 10fundraising Sprint NULL calorie food cart, and 3 others: Creating New Donor Advised Fund Import - https://phabricator.wikimedia.org/T283104 (10Eileenmcnaughton) @MDemosWMF I've just moved this into '... [23:38:40] eileen: is it possile to run the query for T287995 with the large set. i just tried with the default settings and it only gave ~90 results. [23:38:44] T287995: server error seen when attempting to export long search kit list - https://phabricator.wikimedia.org/T287995 [23:39:21] i'm thinking we may be able to get a rough guess by looking at stats for the entry in the logs. [23:39:53] dwisehaupt: there are currently 2 tickets they have open where they have hit the 414 - I guess there are a range of queries they can come up with. I agree that passing ids via url clearly has limits that we are pushing too far - but it will require work to unravel that [23:40:10] you mean a rough guess at how long they are actually trying? [23:43:16] yeah. [23:43:33] i'm thinking that it will show the length of X which is greater than 8k [23:43:43] i could be wrong but it's worth a shot. [23:46:39] yeah I'm sure it's longer than 8k! [23:46:47] the question is how much longer 64k? [23:46:59] & what is the limit that we could extend it to? [23:49:01] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Email: Bulk email export: CiviCRM country should override contribution_tracking country - https://phabricator.wikimedia.org/T152634 (10Ejegg) 05Open→03Resolved a:03Ejegg Current logic only falls back to contribution_tracking.country on empty... [23:49:27] yeah. and we'll have to adjust it in multiple places. checking to see what the hard limits are for each. [23:54:05] doesn't look like there are upper bounds, so in theory we can go as big as we want. :)