[00:01:43] ah, ejegg|food: the clean invalid language is working! [00:01:48] thanks! [00:02:48] (03CR) 10Eileen: [C: 03+2] "Works for me" [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/857809 (owner: 10Ejegg) [00:04:05] 10Fundraising Sprint Incantation optimisation, 10Fundraising Sprint Juggalology 2022, 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, and 2 others: Testmeister esMX email is different from what we have set up in th... - https://phabricator.wikimedia.org/T307704 [00:04:22] wfan: yay [00:04:54] ejegg|food: I think we need a dequeued_rate to go with 'total dequeued' [00:10:51] Hmm - just looking at the latest run [00:10:51] donations_total_messages_dequeued 79 [00:10:52] donations_count{gateway="all"} 49 [00:10:52] donations_processing_rate{period="1s"} 5.4201267162715 [00:10:52] So 38% of them would not 'count' - that would skew the average [00:27:23] 10Fundraising Sprint Incantation optimisation, 10Fundraising Sprint Juggalology 2022, 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, and 2 others: Testmeister esMX email is different from what we have set up in th... - https://phabricator.wikimedia.org/T307704 [00:49:13] eileen: i see some failmail as import _contact error that en_NZ is not an option for field, and it's been cleaned by that WMFDataManagement.CleanInvalidLanguageOptions on purpose, why is en_NZ is added to import contact? [00:49:44] wfan: we should have left en_NZ & made it 'official' - cos it is a real variant? [00:50:02] but if it doesn't exist it should fall back to en_US [00:50:15] while for our WMFDataManagement.CleanInvalidLanguageOptions, we only clean up the name equal label, which en_NZ was not official before [00:50:49] yeah - I think the original phab said 'let's make the real variants officical' - but there is such a trail of them that got lost [00:51:19] But, I think for en_NZ we should add it (I doubt it matters we 'lost' some - since we don't really localise to that) [00:51:43] also if it's will fall back to en_US, we should ignore the import_contact failmail as contact could not be added due to invalid e_NZ ? [00:53:34] and as you said if we want to make en_NZ official, should we create a ticket both upstream and locally? [00:54:51] the one upstream exists - at least one of our phabs starts by saying we should make it official - but we seem to have many [00:55:07] I'm just looking at the language fallback code & it does seem to have a bug somehwere [00:56:24] hmm - by the time it gets to ` if (!wmf_civicrm_check_language_exists($preferredLanguage)) { ` [00:56:35] `preferredLanguage` should be `en_NZ` [00:56:51] & `` $default_locale = Language::getLanguageCode($parts[0]);`` should return `en_US` [00:57:38] ah yeah - at the top of this phab - https://phabricator.wikimedia.org/T321251 - it talks about making languages official [01:02:39] eileen: looks like maybe 60ms total for each discarded duplicate contribution [01:02:53] ejegg: ok - so that would add up [01:03:03] I wonder how much of that is the db query [01:03:27] so i'm just going by the timestamps in the process-control log [01:03:41] how long is it for a non-duplicate - around 150ms? [01:03:49] for the entries where it's just 'Beginning' .. Committing [01:04:11] Let's see, I thought it was 200ms, but maybe less now [01:04:34] oh yeah, around 140ms [01:05:16] ooh, we're doing a lot of normalization before we check for duplicates [01:05:23] I wonder how much of that we can skip [01:06:06] yeah I was thinking the same thing [01:06:39] also - there is a civicrm_initialize in wmf_civicrm_contribution_message_import - maybe that can go - it is outside the timer so it doesn't count though :-) [01:07:26] ah I see we should make en_NZ and en_IN valid for option, is that a way to add them from civi console or we should make a patch to update the language? [01:09:05] wfan: this screen - https://dmaster.localhost:32353/civicrm/admin/options/languages?reset=1 - but before the clean up it wouldn't load cos there were too many languages :-) [01:09:25] (otherwise api explorer if you want to skip doing a patch) [01:12:31] ok, so the query for the dupe check is in the wmf_civicrm_verify_message_and_stage function [01:12:54] which already has a timer - verify_and_stage [01:13:04] let's see what that usually runs [01:13:19] around 5 ms [01:15:48] ok, so I don't want to take away the dupe check inside wmf_civicrm_contribution_message_import [01:16:32] but it might be worth adding another one right up at the start of DonationQueueConsumer::processMessage before even preparing the data for logging or anything [01:17:01] so you are saying verify_and_stage is super quick & not worth skipping - but there is another 55ms to find [01:17:37] yeah, so some bit of that is going to be reading it out of redis [01:18:36] yeah - I still haven't gotten to re-analyse the impact of removing the extraneous `civicrm_initialize()` on the Redis hits - working through the process on that [01:19:10] ok I just added en_NZ via api explore~ this will stop the failmail [01:22:44] (03PS1) 10Ejegg: Try checking for dupes right at the start of import [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857866 [01:22:58] eileen: does that look safe enough to deploy just as a speed test? ^^^ [01:23:23] wfan: [01:23:32] oops, need to reformat that queue consumer class [01:23:46] yay - probably worth trying to replicate locally as the code *should* have given en_US if it didn't exist [01:27:42] (03PS1) 10Ejegg: Reformat Donations Queue Consumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857871 [01:29:01] ejegg: so just to articulate .... [01:29:01] Before the patch it looks like [01:29:01] - _`queue2civicrm_log` [01:29:02] - if `if ( isset( $message['completion_message_id'] ) ) {` do something [01:29:03] - otherwise `wmf_civicrm_check_for_duplicates` [01:29:04] So the patch skips the first logging AND it changes the behaviour if `$message['completion_message_id']` isset [01:29:05] If it were faster it would be because the `queue2civicrm_log` call is slow [01:30:01] yep, that could be the case [01:30:01] (03CR) 10Eileen: [C: 03+2] Reformat Donations Queue Consumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857871 (owner: 10Ejegg) [01:30:05] (03PS2) 10Ejegg: Try checking for dupes right at the start of import [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857866 [01:30:15] ok, rebased on top of reformat [01:30:35] ok - so the behaviour change when `$message['completion_message_id']` isset - when would it be set? [01:31:03] that happens I think for astropay and amazon [01:31:47] where we often get the donation confirmation on an IPN that has precious little donor info [01:32:11] so we have to combine the donation info from the IPN with the donor + utm info from the completion message [01:32:19] (in the pending table) [01:32:44] anyway, I just removed the duplicate check from that else block because it would be redundant, right? [01:32:58] if we switch to always doing it right at the start [01:33:48] So I guess we only have that extra dupe check in that else block is to avoid throwing the MISSING_PREDECESSOR exception [01:34:18] since all donations (including those with completion messages) are checked again for dupes at the end of verify_and_stage [01:40:51] ejegg: so if we always do it at the start we are doing it for completion_message_id when we were not before - if that safe? Do we have enough info to do it? [01:42:05] (03Merged) 10jenkins-bot: Reformat Donations Queue Consumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857871 (owner: 10Ejegg) [01:43:07] eileen: right, so it looks like we assume those two fields exist in all messages [01:43:48] that else clause where we had it before means that there was no pending row found, i.e. we ONLY have the info that came in on the IPN [01:44:00] and we assume that the needed info is there for the dedupe check [01:45:29] ahh, so inside wmf_civicrm_verify_message_and_stage we do check for gateway and gateway_txn_id as required, and throw a different exception if they're not there [01:45:40] I could add that check too [01:46:05] so we make sure to shunt malformed things to the damaged table instead of just dropping them [01:46:29] but... I can't remember the last time we saw something hit the damaged table for that reason, so I'd say it's pretty uncommon [01:49:51] ejegg: so I guess within `wmf_civicrm_check_for_duplicates` maybe check they are not empty & throw exception if so? [01:50:07] ok, could do that [01:50:49] hmm, and change the signature to take the $msg array instead of the two params? [01:52:25] yeah - or just accept e-notices as possibly good noise - they would be empty / NULL [02:13:09] 10Fundraising Sprint Incantation optimisation, 10Fundraising Sprint Juggalology 2022, 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, and 2 others: Testmeister esMX email is different from what we have set up in th... - https://phabricator.wikimedia.org/T307704 [02:14:01] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Still seeing new invalid made up locale saved to contact preferred_language - https://phabricator.wikimedia.org/T323067 (10AnnWF) \Civi\Api4\Action\WMFContact\Save::getPreferredLanguage from this file we can see if we hav... [02:14:19] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Still seeing new invalid made up locale saved to contact preferred_language - https://phabricator.wikimedia.org/T323067 (10AnnWF) 05Open→03Resolved [02:24:14] (03PS1) 10Ejegg: Dupe check fn checks for empty required fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857894 [02:25:26] ok eileen I added that empty check in a separate patch ^^ [02:25:51] I should probably try to understand what's going on in languageland, huh? [02:30:10] ejegg: yeah - I'm not sure if the en_NZ errors were caching related cos the old values just got cleaned up or something we will see more of - I suspect the former [02:32:09] ejegg: so the patches look ok / I'm good to try them - is the check still needed in verify & stage now? Or is it consistently being called before we get to that point? [02:32:47] (03CR) 10Eileen: [C: 03+2] Try checking for dupes right at the start of import [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857866 (owner: 10Ejegg) [02:33:14] (03CR) 10Eileen: [C: 03+2] Dupe check fn checks for empty required fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857894 (owner: 10Ejegg) [02:38:15] eileen: so the dupe check should still be included for other flows that call wmf_civicrm_contribution_message_import [02:38:22] I guess the most common is the file imports [02:39:01] and that calls verify_and_stage but doesn't always go through `process` [02:39:05] I could add an argument to skip it [02:39:06] I guess [02:39:17] it's probably incredibly cheap [02:39:28] yeah, if all of verify_and_stage is 5ms [02:39:48] also it's doing a direct SQL query against a table with indexes on those two columns [02:42:01] oh, those en_NZ donations are all still in the damaged table? [02:42:18] I guess we should requeue them now, if the problem was just cached values? [02:42:20] 10Fundraising Tech - Chaos Crew: Fail Mail (civi1001) run-job: Fetch CiviMail Bounces failed with code - https://phabricator.wikimedia.org/T323057 (10Eileenmcnaughton) @jgleeson @Dwisehaupt @Jgreen - the error is an imap connection error from a lower down library or possibly the imap protocol itself "Could not... [02:44:12] d'oh, now I forget the URL for Dami's new damaged UI [02:45:02] (03Merged) 10jenkins-bot: Try checking for dupes right at the start of import [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857866 (owner: 10Ejegg) [02:45:04] (03Merged) 10jenkins-bot: Dupe check fn checks for empty required fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857894 (owner: 10Ejegg) [02:49:01] ok, I'm going to deploy those and see if they make any difference [02:49:47] (03PS1) 10Eileen: Further fixes on Redis output parsing [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857901 (https://phabricator.wikimedia.org/T322172) [02:49:50] ejegg: cool [02:50:38] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/857902 [02:52:40] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/857902 (owner: 10Ejegg) [03:25:15] (03CR) 10Ejegg: [V: 03+2 C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/857902 (owner: 10Ejegg) [03:27:36] !log civicrm upgraded from 8683d375 to 4b2bc457 [03:27:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:32:31] hmmmmmm no detectable change I think???? [03:33:09] so is the overhead with WmfDatabase::transactionalCall ??? [03:35:15] err, or am I wrong about which of those lines are for duplicate txns [03:35:18] ? [03:36:05] yes I think I am wrong [03:38:05] OK, eileen, sorry, that was a total red herring [03:38:10] I was misinterpreting the logs [03:38:47] that is VERY easily done [03:38:54] Many recurring installments actually have very few lines in those logs, and I was interpreting those as duplicates [03:39:03] which graph are you looking at? [03:39:11] so looking at donations_queue_consume-20221117-032302.log [03:39:20] ah yeah, no graph has the info, sadly [03:39:43] ah right [03:39:47] anyway if you look at that process-control job log and grep for DUPLICATE [03:40:47] you will see that even before that deploy, it was basically taking 5-7ms to get through a duplicate and start the next one [03:41:52] so it's down to 2 now? [03:42:12] eg there are three duplicate messages in a row and their 'already exists' messages are at 03:23:08,105 then 03:23:08,112 then 03:23:08,119 [03:42:22] oh, is it? [03:42:24] looking [03:42:34] well 2022-11-17 03:38:11,159 INFO [03:43:03] seems to end at 2022-11-17 03:38:11,162 ERROR [03:44:06] oh cool. yep, in that file I see up to 5ms, so a tiny improvement [03:44:26] but if you look at https://frmon.frdev.wikimedia.org/d/Pq1YNMviz/fundraising-overview?orgId=1&refresh=1m&from=now-15m&to=now&viewPanel=22 you see it's not really that many duplicates most of the time [03:45:14] dang, it got late here [03:45:27] lol - go to bed :-) [03:45:34] thanks for thinking this through with me eileen ! Have a good rest of your day [03:56:52] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10fr-donorservices: Civi: Forget Me button yielding parameters errors - https://phabricator.wikimedia.org/T322796 (10Ejegg) @MBeat33 We've deployed a fix for this problem and restored the Adyen GDPR calls. Hopefully that's the end of problems with the F... [05:33:30] 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising Sprint Undefined, 10Fundraising-Backlog: Adding and storing rml_source data for RML donations reporting - https://phabricator.wikimedia.org/T321163 (10MSuijkerbuijk_WMF) Yes, this probably needs t... [13:13:01] 10Fundraising Tech - Chaos Crew: Fail Mail (civi1001) run-job: Fetch CiviMail Bounces failed with code - https://phabricator.wikimedia.org/T323057 (10jgleeson) Thanks for digging in @Eileenmcnaughton. I'll add this one to the failmail zoo for future reference. [13:35:15] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops: Process control - would it be possible to log when each job starts & stops - https://phabricator.wikimedia.org/T322170 (10Jgreen) 05Open→03Resolved As far as I can tell, this is done. [13:35:38] yep I think so Jeff_Green. thanks for that one! [13:40:45] 10Fundraising Tech - Chaos Crew: Fail Mail (civi1001) run-job: Fetch CiviMail Bounces failed with code - https://phabricator.wikimedia.org/T323057 (10Jgreen) The thing that stands out to me is that the civi failure message was always within 5-7 seconds of the start of the job. If the timeout is really that short... [13:53:48] 10Fundraising Tech - Chaos Crew: Create documentation for Creating Civi extensions for managing tables on other Databases using Searchkit/Formbuilder for WMFF Civicrm - https://phabricator.wikimedia.org/T322040 (10jgleeson) Nice job on this, I feel like I know a lot more about using SearchKit with FormBuilder as... [13:55:40] 10Fundraising Tech - Chaos Crew: Create documentation for Creating Civi extensions for managing tables on other Databases using Searchkit/Formbuilder for WMFF Civicrm - https://phabricator.wikimedia.org/T322040 (10jgleeson) Also, this is the latest direct link to the docs https://wikitech.wikimedia.org/wiki/Fund... [13:57:13] 10Fundraising Tech - Chaos Crew: Adyen audit sending not refunded recurring ideal donations to the refund queue - https://phabricator.wikimedia.org/T320286 (10jgleeson) a:03jgleeson [14:07:35] 10Fundraising-Backlog: Update DonationStatsCollector to factor in the duplicate donations when calculating processing rates and run time. - https://phabricator.wikimedia.org/T323273 (10jgleeson) [14:09:11] 10Fundraising-Backlog: Update DonationStatsCollector to factor in the duplicate donations when calculating processing rates and run time. - https://phabricator.wikimedia.org/T323273 (10jgleeson) The patch to show us the total processed, excluding duplicates is here https://phabricator.wikimedia.org/T323273 [14:13:35] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Review our queue speed - is it OK? - https://phabricator.wikimedia.org/T322791 (10jgleeson) I've created {T323273} as a follow on to @Ejegg's dequeue stats patch [14:14:53] 10Fundraising-Backlog: Update DonationStatsCollector to factor in the duplicate donations when calculating processing rates and run time. - https://phabricator.wikimedia.org/T323273 (10jgleeson) [14:15:08] 10Fundraising-Backlog: Update DonationStatsCollector to factor in the duplicate donations when calculating processing rates and run time. - https://phabricator.wikimedia.org/T323273 (10jgleeson) [14:15:21] 10Fundraising-Backlog: Update DonationStatsCollector to factor in the duplicate donations when calculating processing rates and run time. - https://phabricator.wikimedia.org/T323273 (10jgleeson) [15:25:59] 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising Sprint Undefined, 10Fundraising-Backlog: Adding and storing rml_source data for RML donations reporting - https://phabricator.wikimedia.org/T321163 (10EWilfong_WMF) Agreed that this will be a long... [15:45:49] (03PS11) 10Jgleeson: CreateRecurringPaymentsProfile [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) (owner: 10Wfan) [15:46:52] just rebasing ^ [16:14:38] (03CR) 10Jgleeson: [C: 04-1] "Thanks for moving the tests around. This patch is looking really good! I've left a couple of suggestions inline." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) (owner: 10Wfan) [16:15:48] -1 always feels harsh when it's just small things but 0 feels like it doesn't require changes so I go with the heavy handed -1 [16:40:49] 10Fundraising Tech - Chaos Crew: Create documentation for Creating Civi extensions for managing tables on other Databases using Searchkit/Formbuilder for WMFF Civicrm - https://phabricator.wikimedia.org/T322040 (10Damilare) Thanks @jgleeson for reviewing the doc, I've made changes to those parts and also added a... [16:42:11] thanks damilare. might be worth updating the URL in https://phabricator.wikimedia.org/T322040#8357440 [16:42:35] I dropped the working link in because clicking that link in your post didn't take me to the new docs [16:44:28] I added the new links to the topics under that comment [16:44:45] but I've also updated the old link too [16:45:09] Also thanks for catching that [16:45:10] yeah it looks like the structure changed during the task [16:45:25] your links work great too [16:46:10] I have a feeling those docs will be getting referred to a lot as we move more things over to searchkit forms [16:47:45] ooh niice damilare but do you mind if I move those new pages into our fundraising namespace? wikispace? not sure the right word [16:47:57] oh yh cstone did some restructuring on that earlier. [16:48:27] sure thing cstone, not sure what a wikispace is also lol [16:48:39] yeah i dont know the right worde [16:48:42] word* [16:48:54] but it would be like fundraising/civicrm/{your new pages} [16:49:01] instead of just {your new pages} [16:51:01] right that makes sense, thanks for catching that [16:59:57] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10fr-donorservices: Civi: Forget Me button yielding parameters errors - https://phabricator.wikimedia.org/T322796 (10MBeat33) 05Open→03Resolved a:03MBeat33 Awesome, thank you @Ejegg I tested on a few and it looks great. [17:01:12] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10fr-donorservices: Civi: Forget Me button yielding parameters errors - https://phabricator.wikimedia.org/T322796 (10greg) 05Resolved→03Open (just reopening so we can review during our sprint review next week, glad all is good!) [17:01:18] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Still seeing new invalid made up locale saved to contact preferred_language - https://phabricator.wikimedia.org/T323067 (10greg) 05Resolved→03Open (just reopening so we can review during sprint review) [19:29:36] (03Abandoned) 10Thiemo Kreuz (WMDE): Replace obsolete HTML with CSS [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/266064 (https://phabricator.wikimedia.org/T108259) (owner: 10MtDu) [19:39:13] cstone: is now a good time to chat? [19:39:59] sure Jeff_Green [19:40:04] bah jgleeson [19:40:07] sorry jeff [19:40:10] ha! [19:40:22] ha [19:42:31] how about today's standup link? [19:42:34] sounds good [19:45:49] 10Wikimedia-Fundraising-Banners, 10Content-Transform-Team-WIP, 10Wikipedia-iOS-App-Backlog, 10ios-app-v7.0: 2022 English fundraising messages in iOS app - https://phabricator.wikimedia.org/T322950 (10LGoto) a:03Seddon [19:59:29] 10Fundraising-Backlog: Requesting access to enable/disable CentralNotice for WFan (WMF) - https://phabricator.wikimedia.org/T323313 (10AnnWF) [20:05:33] 10fundraising-tech-ops: Fundraising access request for spuri - https://phabricator.wikimedia.org/T323314 (10Dwisehaupt) [20:06:29] 10fundraising-tech-ops: Fundraising access request for spuri - https://phabricator.wikimedia.org/T323314 (10Dwisehaupt) Approval. `Date: Tue, 15 Nov 2022 14:28:16 From: Lisa Seitz Gruwell To: Megan Hernandez Cc: Erica Roden, Sheetal Puri, Dallas Wisehaupt Subject: Re: Lisa - please approve Sheetal civi access... [20:10:31] 10Fundraising-Backlog, 10Trust-and-Safety: Requesting access to enable/disable CentralNotice for WFan (WMF) - https://phabricator.wikimedia.org/T323313 (10AnnWF) [20:12:01] 10Fundraising-Backlog, 10Trust-and-Safety: Requesting access to enable/disable CentralNotice for WFan (WMF) - https://phabricator.wikimedia.org/T323313 (10AnnWF) [20:13:20] 10Fundraising-Backlog, 10Trust-and-Safety: Requesting access to enable/disable CentralNotice for WFan (WMF) - https://phabricator.wikimedia.org/T323313 (10greg) @AnnWF would need to be added to the centralnoticeadmin group, afaict: https://meta.wikimedia.org/w/index.php?title=Special:ListUsers&group=centralnot... [20:16:39] (03PS1) 10Ejegg: Avoid enotice on contacts with no wmf_donor data [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/858411 [20:28:15] cstone were you working with eileen on the redis log parsing stuff? [20:28:39] I'm trying to review her latest patch on it, and I'm not sure I've got my redis monitor output in the format she expects [20:28:49] ejegg: I didn't besides +2ing stuff she had already confirmed [20:28:56] ah ok [20:31:35] (03CR) 10Ejegg: "Could you add a bit more info (maybe in the API class comment) on how to create the redis log for parsing? I tried just redirecting the ./" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857901 (https://phabricator.wikimedia.org/T322172) (owner: 10Eileen) [20:34:52] 10Fundraising Tech - Chaos Crew: Adyen audit sending not refunded recurring ideal donations to the refund queue - https://phabricator.wikimedia.org/T320286 (10jgleeson) **Queries scratchpad:** ` SELECT cr.* FROM civicrm.log_civicrm_contribution_recur cr INNER JOIN civicrm.log_civicrm_contribution_rec... [21:11:28] (03PS12) 10Wfan: CreateRecurringPaymentsProfile [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) [21:13:50] (03PS4) 10Wfan: Add error handling and mapping error for Smashpig PayPal EC GetExpressCheckoutDetails API call [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/857150 (https://phabricator.wikimedia.org/T322570) [21:14:31] 10Fundraising Tech - Chaos Crew: Adyen audit changing the contribution_recur currency to USD in a chargeback (maybe) - https://phabricator.wikimedia.org/T320286 (10Cstone) [21:31:24] (03CR) 10Eileen: "ejegg - the notes are in our docs - https://wikitech.wikimedia.org/wiki/Fundraising/Internal-facing/CiviCRM#Redis_monitoring" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857901 (https://phabricator.wikimedia.org/T322172) (owner: 10Eileen) [21:32:45] (03CR) 10Ejegg: "Thanks for all this work! I've got a difference of opinion on how to handle the case of a totally broken response, as well as some other s" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) (owner: 10Wfan) [21:33:23] (03CR) 10Eileen: "I didn't know about the script you are referring to - but since the goal is to parse output from prod it's best to leave it out of it - si" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857901 (https://phabricator.wikimedia.org/T322172) (owner: 10Eileen) [21:35:54] (03CR) 10Eileen: [C: 03+2] "I'm pretty sure my IDE would complain that a more expensive condition is after the cheaper one in the IF (I use the EA Extended plugin htt" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/858411 (owner: 10Ejegg) [21:36:02] (03CR) 10Ejegg: Further fixes on Redis output parsing (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857901 (https://phabricator.wikimedia.org/T322172) (owner: 10Eileen) [21:48:31] (03Merged) 10jenkins-bot: Avoid enotice on contacts with no wmf_donor data [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/858411 (owner: 10Ejegg) [22:00:44] (03PS1) 10Eileen: Hack imap timeout to 15 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/858431 (https://phabricator.wikimedia.org/T323057) [22:01:21] 10Fundraising Tech - Chaos Crew: Fail Mail (civi1001) run-job: Fetch CiviMail Bounces failed with code - https://phabricator.wikimedia.org/T323057 (10Eileenmcnaughton) Let's try this then - https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/858431 - if it works then I'll work on a 'real' fix for it [22:02:15] 10fundraising-tech-ops: Fundraising access request for spuri - https://phabricator.wikimedia.org/T323314 (10Dwisehaupt) [22:03:21] 10fundraising-tech-ops: Fundraising access request for spuri - https://phabricator.wikimedia.org/T323314 (10Dwisehaupt) a:03Dwisehaupt SSL client cert created and sent. Password sent via SMS. Civi and superset accounts created and set with temp passwords. Instructions sent on how to log in and change those pas... [22:06:15] 10Fundraising Tech - Chaos Crew: Fail Mail (civi1001) run-job: Fetch CiviMail Bounces failed with code - https://phabricator.wikimedia.org/T323057 (10Eileenmcnaughton) Apparently this is the place in the code https://github.com/civicrm/civicrm-core/blob/master/CRM/Mailing/MailStore/Imap.php#L53 [22:17:42] 10Fundraising-Analysis, 10Fundraising-Backlog, 10FR-Email: Diff Tool to compare two emails - Test vs Control - https://phabricator.wikimedia.org/T322891 (10greg) Just covering our bases here: Have you reached out to Acoustic to ask if this is a feature they can provide? [23:51:39] 10Fundraising Tech - Chaos Crew, 10Documentation: Complete Adyen Checkout documentation to include donation statuses for all stages/permutations - https://phabricator.wikimedia.org/T316583 (10greg) [23:51:41] 10Fundraising Tech - Chaos Crew, 10Documentation: Create documentation listing FR-tech service providers and URL - https://phabricator.wikimedia.org/T316118 (10greg) [23:51:43] 10Fundraising Tech - Chaos Crew, 10Documentation: Document which tables are hit by each process-control job - https://phabricator.wikimedia.org/T322243 (10greg) [23:51:45] 10Fundraising Tech - Chaos Crew, 10Documentation: Documentation: Banner data flow and data stores - https://phabricator.wikimedia.org/T296222 (10greg) [23:58:08] 10Fundraising Tech - Chaos Crew: Deleted Contact Overwriting Current Contact Record - https://phabricator.wikimedia.org/T311438 (10greg) @Jsimmonds Based on Eileen's last comment ( https://phabricator.wikimedia.org/T311438#8398618 ) is this still an issue? Sorry about the delay in responding here!