[00:15:26] 10Fundraising Tech - Chaos Crew: Deleted Contact Overwriting Current Contact Record - https://phabricator.wikimedia.org/T311438 (10greg) p:05Triage→03Medium [00:15:43] 10Fundraising Tech - Chaos Crew: Document which tables are hit by each process-control job - https://phabricator.wikimedia.org/T322243 (10greg) p:05Triage→03Low [00:15:54] 10Fundraising Tech - Chaos Crew: Documentation: Banner data flow and data stores - https://phabricator.wikimedia.org/T296222 (10greg) p:05Triage→03Low [00:16:08] 10Fundraising Tech - Chaos Crew: Create documentation listing FR-tech service providers and URL - https://phabricator.wikimedia.org/T316118 (10greg) p:05Triage→03Medium [00:16:25] 10Fundraising Tech - Chaos Crew: Create documentation for Creating Civi extensions for managing tables on other Databases using Searchkit/Formbuilder for WMFF Civicrm - https://phabricator.wikimedia.org/T322040 (10greg) p:05Triage→03Medium [00:30:07] (03CR) 10Wfan: [C: 03+2] Add Paypal EC PaymentProvider Tests [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/855749 (https://phabricator.wikimedia.org/T318756) (owner: 10Jgleeson) [00:30:54] (03Merged) 10jenkins-bot: Add Paypal EC PaymentProvider Tests [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/855749 (https://phabricator.wikimedia.org/T318756) (owner: 10Jgleeson) [01:13:14] (03PS10) 10Wfan: CreateRecurringPaymentsProfile [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) [01:14:00] (03CR) 10Wfan: CreateRecurringPaymentsProfile (031 comment) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) (owner: 10Wfan) [01:20:56] (03PS1) 10Eileen: Fix 745c4241a7 to really add fastArray cache to mixin caches [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857124 (https://phabricator.wikimedia.org/T322172) [01:21:56] cstone: it turns out my first cache fix did nothing cos there was a code fix missing AND I needed to clear some more cache stuff - here is it the code fix part.. https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/857124 [01:23:41] ahh ok [01:36:26] (03PS1) 10Eileen: Remove civicrm_initialize from thank_you_for_contribution [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857137 (https://phabricator.wikimedia.org/T322172) [01:37:00] (03CR) 10Cstone: [C: 03+2] Fix 745c4241a7 to really add fastArray cache to mixin caches [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857124 (https://phabricator.wikimedia.org/T322172) (owner: 10Eileen) [01:37:31] I think I just spent have a day coming up with a minus-one-line patch :-) [01:37:49] aw hah its not the amount of code its the journey :P [01:38:51] :-) [01:41:19] (03PS1) 10Eileen: Add code I used to analyse the Redis Montior output [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857138 (https://phabricator.wikimedia.org/T322172) [01:41:55] cstone: ah well I just committed the journey then ^^ :-) [01:47:00] i need to dig in more on the prometheus injestion. i was pretty sure i tested the ability to ingest with tighter timeframes than 1 min injestion. [01:47:24] but that was probably around this time last year and has totally popped off the mental stack. [01:49:18] (03Merged) 10jenkins-bot: Fix 745c4241a7 to really add fastArray cache to mixin caches [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857124 (https://phabricator.wikimedia.org/T322172) (owner: 10Eileen) [01:50:30] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/857139 [01:50:43] wfan: FYI - rolling out your is_deleted patch [01:51:07] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/857139 (owner: 10Eileen) [01:59:26] oho my local template finally updated like 4 days later [01:59:47] thank you email template* [02:00:45] (03CR) 10Cstone: [C: 03+2] "Nice find!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857137 (https://phabricator.wikimedia.org/T322172) (owner: 10Eileen) [02:01:06] eileen: are there probably other places civicrm_initalize is being called too much then? [02:02:25] (03CR) 10Cstone: [C: 03+2] Add code I used to analyse the Redis Montior output [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857138 (https://phabricator.wikimedia.org/T322172) (owner: 10Eileen) [02:05:33] (03PS1) 10Wfan: Add error handling and mapping error for Smashpig PayPal EC GetExpressCheckoutDetails API call [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/857150 (https://phabricator.wikimedia.org/T322570) [02:06:18] (03CR) 10CI reject: [V: 04-1] Add error handling and mapping error for Smashpig PayPal EC GetExpressCheckoutDetails API call [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/857150 (https://phabricator.wikimedia.org/T322570) (owner: 10Wfan) [02:10:37] (03PS2) 10Wfan: Add error handling and mapping error for Smashpig PayPal EC GetExpressCheckoutDetails API call [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/857150 (https://phabricator.wikimedia.org/T322570) [02:11:02] (03CR) 10CI reject: [V: 04-1] Add error handling and mapping error for Smashpig PayPal EC GetExpressCheckoutDetails API call [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/857150 (https://phabricator.wikimedia.org/T322570) (owner: 10Wfan) [02:13:00] (03Merged) 10jenkins-bot: Remove civicrm_initialize from thank_you_for_contribution [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857137 (https://phabricator.wikimedia.org/T322172) (owner: 10Eileen) [02:13:53] (03PS3) 10Wfan: Add error handling and mapping error for Smashpig PayPal EC GetExpressCheckoutDetails API call [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/857150 (https://phabricator.wikimedia.org/T322570) [02:14:36] (03Merged) 10jenkins-bot: Add code I used to analyse the Redis Montior output [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857138 (https://phabricator.wikimedia.org/T322172) (owner: 10Eileen) [03:21:51] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10fr-donorservices: Civi: Forget Me button yielding parameters errors - https://phabricator.wikimedia.org/T322796 (10Ejegg) Oh wow @Eileenmcnaughton, it really is api3 making a bad query: `lang=sql SELECT civicrm_contribution.id, contact_a.id as contac... [03:39:55] (03PS1) 10Ejegg: Revert "Remove Adyen GDPR calls for now" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857230 (https://phabricator.wikimedia.org/T322796) [03:39:57] (03PS1) 10Ejegg: Fix Adyen GDPR forgetme calls for merged contacts [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857231 (https://phabricator.wikimedia.org/T322796) [03:46:25] !log civicrm upgraded from fa71f219 to 85c98fc7 [03:46:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:04:45] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Oh Behave Dedupe - https://phabricator.wikimedia.org/T323081 (10Eileenmcnaughton) The patch is deployed - but we should monitor for a bit [04:05:03] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Excessive Redis cache hits - https://phabricator.wikimedia.org/T322172 (10Eileenmcnaughton) I just updated docs https://wikitech.wikimedia.org/wiki/Fundraising/Internal-facing/CiviCRM#Redis_monitoring [04:06:20] (03CR) 10Eileen: [C: 03+2] Fix Adyen GDPR forgetme calls for merged contacts [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857231 (https://phabricator.wikimedia.org/T322796) (owner: 10Ejegg) [04:07:40] thanks eileen! [04:15:41] (03CR) 10AndyRussG: [C: 03+2] "Cool!!! Thanks!! :)" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/856617 (owner: 10Jgleeson) [04:16:14] (03Merged) 10jenkins-bot: Update Adyen and Ingenico to use IGetLatestPaymentStatusProvider. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/856617 (owner: 10Jgleeson) [04:33:05] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Review our queue speed - is it OK? - https://phabricator.wikimedia.org/T322791 (10Eileenmcnaughton) There might be something in the mix that is slow & longer ones have more chance of hitting it? [04:59:55] 10Fundraising Tech - Chaos Crew: Deleted Contact Overwriting Current Contact Record - https://phabricator.wikimedia.org/T311438 (10Eileenmcnaughton) On digging into this I found that my Acoustic account is disabled :-( However, on reviewing is within CiviCRM https://civicrm.wikimedia.org/civicrm/a/#/omnimail/re... [05:14:22] 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Decide a naming convention for our extensions & document on wikitech - https://phabricator.wikimedia.org/T321638 (10Eilee... [05:46:50] 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising Sprint Undefined, 10Fundraising-Backlog: Adding and storing rml_source data for RML donations reporting - https://phabricator.wikimedia.org/T321163 (10Eileenmcnaughton) We discussed this today & g... [09:00:37] 10Fundraising-Backlog: Unsubscribe link broken - https://phabricator.wikimedia.org/T323197 (10krobinson) [09:01:48] 10Fundraising-Backlog: Unsubscribe link broken - https://phabricator.wikimedia.org/T323197 (10krobinson) p:05Triage→03Unbreak! [09:02:15] 10Fundraising-Backlog: Unsubscribe link broken - https://phabricator.wikimedia.org/T323197 (10krobinson) [09:08:49] 10Fundraising-Backlog: Unsubscribe link broken - https://phabricator.wikimedia.org/T323197 (10krobinson) [09:09:54] 10Fundraising-Backlog: Unsubscribe link broken - https://phabricator.wikimedia.org/T323197 (10krobinson) [09:26:48] 10Fundraising-Backlog: Unsubscribe link broken - https://phabricator.wikimedia.org/T323197 (10krobinson) [11:00:13] 10Fundraising-Backlog: Unsubscribe link broken - https://phabricator.wikimedia.org/T323197 (10Pcoombe) It looks to me like the page is available again [11:04:23] 10Fundraising-Backlog: Unsubscribe link broken - https://phabricator.wikimedia.org/T323197 (10Pcoombe) The unsubscribe page is hosted by Acoustic. According to https://status.goacoustic.com/ they have been seeing issues this morning on Pod 4, which I think is the one we use [12:34:20] ok fr-tech it looks like we have an unbreak now [12:34:37] Kristie messaged on slack about this https://phabricator.wikimedia.org/T323197 [12:35:43] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Unsubscribe link broken - https://phabricator.wikimedia.org/T323197 (10jgleeson) [12:36:42] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Unsubscribe link broken - https://phabricator.wikimedia.org/T323197 (10jgleeson) a:03jgleeson [12:37:24] hmmm ok that URL is loading for me [12:40:56] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Unsubscribe link broken - https://phabricator.wikimedia.org/T323197 (10jgleeson) ah, thanks, @Pcoombe. I missed your comment. I can also confirm the URL is working for me. [12:44:22] ok looks like that one is resolved for now. Acoustic, which hosts the unsubscribe form, looks like it had some issues that have now been resolved [12:46:21] well that was a good panic ticket to get the blood going and start the day [13:10:06] 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising Sprint Undefined, 10Fundraising-Backlog: Adding and storing rml_source data for RML donations reporting - https://phabricator.wikimedia.org/T321163 (10MSuijkerbuijk_WMF) @EWilfong_WMF adding you t... [15:22:28] 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising Sprint Undefined, 10Fundraising-Backlog: Adding and storing rml_source data for RML donations reporting - https://phabricator.wikimedia.org/T321163 (10EWilfong_WMF) Hi all, @Eileenmcnaughton, I t... [15:28:18] 10Fundraising Sprint Incantation optimisation, 10Fundraising Sprint Juggalology 2022, 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, and 2 others: Testmeister esMX email is different from what we have set up in th... - https://phabricator.wikimedia.org/T307704 [15:31:36] 10Fundraising Sprint Incantation optimisation, 10Fundraising Sprint Juggalology 2022, 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, and 2 others: Testmeister esMX email is different from what we have set up in th... - https://phabricator.wikimedia.org/T307704 [16:29:06] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Unsubscribe link broken - https://phabricator.wikimedia.org/T323197 (10greg) 05Open→03Invalid Marking this as invalid (since it wasn't actually an issue on our end, but with Acoustic) so as to not keep around an UBN! open in phab (many folks watch for... [17:06:50] (03CR) 10Cstone: [C: 03+2] Revert "Remove Adyen GDPR calls for now" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857230 (https://phabricator.wikimedia.org/T322796) (owner: 10Ejegg) [17:20:14] (03Merged) 10jenkins-bot: Revert "Remove Adyen GDPR calls for now" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857230 (https://phabricator.wikimedia.org/T322796) (owner: 10Ejegg) [17:20:16] (03Merged) 10jenkins-bot: Fix Adyen GDPR forgetme calls for merged contacts [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857231 (https://phabricator.wikimedia.org/T322796) (owner: 10Ejegg) [18:01:13] 10Fundraising Tech - Chaos Crew: Deleted Contact Overwriting Current Contact Record - https://phabricator.wikimedia.org/T311438 (10KHaggard) @Eileenmcnaughton - I just re-enabled your account in Acoustic. Not sure why it keeps disabling - maybe password expiring? Anyway, let me know if you have any issues loggin... [19:36:54] (03CR) 10Jgleeson: Export donations qc stats every minute (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857074 (https://phabricator.wikimedia.org/T323162) (owner: 10Ejegg) [19:51:03] 10Fundraising Sprint Incantation optimisation, 10Fundraising Sprint Juggalology 2022, 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, and 2 others: Testmeister esMX email is different from what we have set up in th... - https://phabricator.wikimedia.org/T307704 [20:09:28] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Review our queue speed - is it OK? - https://phabricator.wikimedia.org/T322791 (10jgleeson) Ok this graph does a better job of showing how things are slowing down as the volume picks up https://frmon.frdev.wikimedia.org/d... [20:26:02] 10Fundraising Sprint Incantation optimisation, 10Fundraising Sprint Juggalology 2022, 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, and 2 others: Testmeister esMX email is different from what we have set up in th... - https://phabricator.wikimedia.org/T307704 [20:27:53] How do we pull a phab back into the triage column - do we have to 'find' it & drag it ( https://phabricator.wikimedia.org/T307704 is getting pings) [20:29:43] 10Fundraising Tech - Chaos Crew: Deleted Contact Overwriting Current Contact Record - https://phabricator.wikimedia.org/T311438 (10Eileenmcnaughton) thanks @KHaggard [20:33:27] it should be there now eileen https://phabricator.wikimedia.org/tag/fundraising-backlog/ [20:33:57] 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising Sprint Undefined, 10Fundraising-Backlog: Adding and storing rml_source data for RML donations reporting - https://phabricator.wikimedia.org/T321163 (10Eileenmcnaughton) @EWilfong_WMF I was going o... [20:34:13] thanks jgleeson [20:34:18] np [20:46:17] 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising Sprint Undefined, 10Fundraising-Backlog: Adding and storing rml_source data for RML donations reporting - https://phabricator.wikimedia.org/T321163 (10Ejegg) So this would mean changes in every co... [20:48:06] greg-g: just chatting with ejegg about https://phabricator.wikimedia.org/T321163 & our feeling is it might not be something we should do at this time of year - but maybe wondering if you could be the one to say so.... [20:50:26] 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising Sprint Undefined, 10Fundraising-Backlog: Adding and storing rml_source data for RML donations reporting - https://phabricator.wikimedia.org/T321163 (10EWilfong_WMF) Yes, I was proposing adding a n... [21:10:22] eileen: gotcha, reading [21:31:13] 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising Sprint Undefined, 10Fundraising-Backlog: Adding and storing rml_source data for RML donations reporting - https://phabricator.wikimedia.org/T321163 (10greg) >>! In T321163#8400739, @Ejegg wrote: >... [22:03:55] (03CR) 10Wfan: [C: 03+2] Add name to wmffraud info file [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857086 (owner: 10Eileen) [22:17:05] (03Merged) 10jenkins-bot: Add name to wmffraud info file [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857086 (owner: 10Eileen) [22:23:05] 10Fundraising Sprint Incantation optimisation, 10Fundraising Sprint Juggalology 2022, 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, and 2 others: Testmeister esMX email is different from what we have set up in th... - https://phabricator.wikimedia.org/T307704 [22:35:58] (03PS1) 10Ejegg: Add prometheus stat for total messages dequeued [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857791 [22:38:34] 10Fundraising Sprint Incantation optimisation, 10Fundraising Sprint Juggalology 2022, 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, and 2 others: Testmeister esMX email is different from what we have set up in th... - https://phabricator.wikimedia.org/T307704 [22:43:34] 10Fundraising Sprint Incantation optimisation, 10Fundraising Sprint Juggalology 2022, 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, and 2 others: Testmeister esMX email is different from what we have set up in th... - https://phabricator.wikimedia.org/T307704 [22:47:17] (03PS2) 10Ejegg: Add prometheus stat for total messages dequeued [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857791 [22:53:29] (03PS3) 10Ejegg: Add prometheus stat for total messages dequeued [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857791 [22:53:51] (03CR) 10Jgleeson: [C: 03+2] "LGTM!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857791 (owner: 10Ejegg) [23:07:05] (03Merged) 10jenkins-bot: Add prometheus stat for total messages dequeued [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857791 (owner: 10Ejegg) [23:18:58] thanks jgleeson [23:19:07] ok, shall we do that civi deploy now? [23:19:15] I'll prep it [23:19:48] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/857803 [23:21:23] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/857803 (owner: 10Ejegg) [23:25:22] (03PS1) 10Ejegg: Enable Redis cache backend for CiviCRM [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/857809 [23:25:44] ok eileen ^^^ seems to get redis caching enabled for me locally [23:29:05] going to do that deploy [23:29:13] I guess we should keep an eye on the TY mail sender [23:37:23] !log civicrm upgraded from 85c98fc7 to 8683d375 [23:37:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:38:26] ok, looks like it's still sending TY messages just fine [23:39:41] the run after that deploy was actually a bit slower than the previous one, but there's a lot of variation [23:47:16] cool, the donations_total_messages_dequeued stat is showing up in the .prom files [23:47:20] let's add it to the graph! [23:53:30] ok, it's on there [23:53:48] I wonder if '% of messages that are duplicates' would be interesting? [23:53:55] hmm, food time