[00:13:03] 10Fundraising-Backlog, 10fr-donorservices: Adyen donors think they are recurring July 2022 - https://phabricator.wikimedia.org/T313854 (10Cstone) This is confusing so I started some documentation here: https://wikitech.wikimedia.org/wiki/Fundraising/Data_and_flow/PSP_integrations/Adyen_Checkout#Subscription_Mo... [00:21:10] 10Fundraising-Backlog, 10fr-donorservices: Adyen donors think they are recurring July 2022 - https://phabricator.wikimedia.org/T313854 (10EMartin) @Cstone I received a one-time TY email as well. {F35349544}. @MBeat33 What is it donors are seeing to think their donation is recurring? [01:05:59] 10Fundraising-Backlog, 10fr-donorservices: Adyen donors think they are recurring July 2022 - https://phabricator.wikimedia.org/T313854 (10Ejegg) @EMartin Some credit card companies show you who has your card saved. I don't think this is a problem with Adyen - we tell them to tokenize every transaction when MC... [01:14:04] 10Fundraising-Backlog, 10fr-donorservices: Adyen donors think they are recurring July 2022 - https://phabricator.wikimedia.org/T313854 (10Ejegg) Actually, based on this explanation from my card company's 'Recurring Charges' page, even deleting the Adyen-side token with the API call might not get us off that li... [01:22:39] (03PS1) 10Ejegg: Revert "Debug logging for queue speed investigation" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/817917 [01:22:41] (03CR) 10Ejegg: [C: 03+2] Revert "Debug logging for queue speed investigation" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/817917 (owner: 10Ejegg) [01:32:42] 10Fundraising-Backlog, 10fr-donorservices: Adyen donors think they are recurring July 2022 - https://phabricator.wikimedia.org/T313854 (10EMartin) @Ejegg Adyen is assigning someone to talk to about this. I'll reach out when I have that person and run your questions by them and loop you in. Good that there ar... [01:36:35] (03Merged) 10jenkins-bot: Revert "Debug logging for queue speed investigation" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/817917 (owner: 10Ejegg) [02:05:00] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Investigate queue consumer slowdown in pretest of June 13 2022 - https://phabricator.wikimedia.org/T313000 (10Ejegg) So we enabled some extra debug logging for some of today's test, and also looked more closely at the timers we've had all along. Our avera... [03:21:03] 10Fundraising Sprint Machine Unlearning, 10Fundraising-Backlog, 10FR-Alerts, 10FR-Smashpig, 10MediaWiki-extensions-DonationInterface: Don't send failmail on Adyen unsupported card - https://phabricator.wikimedia.org/T313069 (10Ejegg) Hmm, we're still getting failmail on these errors. I guess it wasn't as... [03:21:22] (03PS1) 10Ejegg: Update donation-interface and smash-pig [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/817921 [03:21:29] (03CR) 10Ejegg: [C: 03+2] Update donation-interface and smash-pig [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/817921 (owner: 10Ejegg) [03:23:18] (03CR) 10Ejegg: [V: 03+2 C: 03+2] Update donation-interface and smash-pig [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/817921 (owner: 10Ejegg) [03:26:29] (03PS1) 10Ejegg: Revert "Debug logging for queue speed investigation" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/817922 [03:26:33] (03CR) 10Ejegg: [C: 03+2] Revert "Debug logging for queue speed investigation" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/817922 (owner: 10Ejegg) [03:28:56] !log updated fundraising CiviCRM from e0962be6 to 3143dda9 [03:28:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:07:12] PROBLEM - check_procs on frdb2003 is CRITICAL: PROCS CRITICAL: 2219 processes https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_procs [05:17:12] RECOVERY - check_procs on frdb2003 is OK: PROCS OK: 246 processes https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_procs [08:16:14] (03CR) 10Abijeet Patro: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_38) - 10https://gerrit.wikimedia.org/r/817483 (owner: 10L10n-bot) [09:14:18] 10Fundraising Sprint Machine Unlearning, 10Fundraising-Backlog: 3DS needed on CarteBaincaire in France - https://phabricator.wikimedia.org/T313528 (10krobinson) Unfortunately this doesn't seem to be an issue of the past: we've seen donors reporting errors since the email send an hour ago, and 800 declines all... [09:46:35] 10Fundraising Sprint Machine Unlearning, 10Fundraising-Backlog: 3DS needed on CarteBaincaire in France - https://phabricator.wikimedia.org/T313528 (10RKumar_WMF) I ran a report and do see 350+ Carte Bancaire transactions in Refused status in the Adyen console. The reason code is Acquirer response 3D Not Authe... [09:51:02] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Investigate queue consumer slowdown in pretest of June 13 2022 - https://phabricator.wikimedia.org/T313000 (10Eileenmcnaughton) Hmm interesting - that was a moderated version jump which makes it harder to bi-sect but will ponder what changed [12:29:57] (03PS4) 10Damilare Adedoyin: Update PayPal IPN listener to discard donations initiated via Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/817733 (https://phabricator.wikimedia.org/T313512) [12:30:39] Thanks wfan šŸ¤¦ā€ā™‚ļø [12:34:26] (03PS5) 10Damilare Adedoyin: Update PayPal IPN listener to discard donations initiated via Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/817733 (https://phabricator.wikimedia.org/T313512) [12:35:06] (03CR) 10Damilare Adedoyin: "Thanks Elliot for the comments" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/817733 (https://phabricator.wikimedia.org/T313512) (owner: 10Damilare Adedoyin) [15:05:28] 10Fundraising-Backlog, 10fr-donorservices: Adyen donors think they are recurring July 2022 - https://phabricator.wikimedia.org/T313854 (10MBeat33) > What is it donors are seeing to think their donation is recurring if it is actually show up as one time everywhere but the Adyen console? @EMartin I am gathering... [15:40:01] (03CR) 10Wfan: [C: 03+2] "Looks really good to me and since Elliott is not here today, would love to plus 2 for that to do a frdev testing soon, since you went thro" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/817733 (https://phabricator.wikimedia.org/T313512) (owner: 10Damilare Adedoyin) [15:40:45] (03Merged) 10jenkins-bot: Update PayPal IPN listener to discard donations initiated via Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/817733 (https://phabricator.wikimedia.org/T313512) (owner: 10Damilare Adedoyin) [15:43:45] 10Fundraising-Backlog, 10fr-donorservices: Adyen donors think they are recurring July 2022 - https://phabricator.wikimedia.org/T313854 (10EMartin) From Adyen thread FRTech is now looped in on: //The reason your shoppers are seeing 'recurring' on their one-time payments is because you are specifying in your pa... [15:51:46] Thanks wFan for the CR [15:53:20] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Investigate queue consumer slowdown in pretest of July 13 2022 - https://phabricator.wikimedia.org/T313000 (10Cstone) [15:58:16] hmm [15:58:40] that adyen reply might make us need to rethink monthlyconvert cstone [16:00:19] https://github.com/wikimedia/mediawiki-extensions-DonationInterface/blob/00c178bbcf755af3ce666dcca44ecb3a6e019247/adyen_gateway/adyen_checkout.adapter.php#L43 [16:00:31] well we purposely set it up that way i thought? [16:00:54] ya [16:21:57] sorry fr-tech [16:22:13] no worries jgleeson [16:22:19] it's ok jgleeson :) everything ok there? [16:22:19] one of the builders who did work on the house turned up unexpected to finish something off [16:22:42] all good thanks [16:24:01] my standup was that I've spent a bit of time digging into the adyen logs due to the 3DS emails and then I've also been fixing some bugs in the WIP patches I pushed up for the braintree IPN stuff which I'm still working on [16:26:50] 10Fundraising-Backlog, 10fr-donorservices: Adyen donors think they are recurring July 2022 - https://phabricator.wikimedia.org/T313854 (10Cstone) @EMartin we have it set up that way for monthly convert. If we don't tokenize the first payment details we would have to ask again for the credit card info when sett... [17:13:13] (03PS1) 10Damilare Adedoyin: Update smashpig from v0.8.2 to v0.8.3 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/818185 [17:15:47] Thanks dami for the smash pig upgrade :) [17:23:09] (03CR) 10Damilare Adedoyin: [C: 03+2] Update smashpig from v0.8.2 to v0.8.3 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/818185 (owner: 10Damilare Adedoyin) [17:23:30] my pleasure wfan, can we do the deployment to payment-staging together? [17:23:54] Sure! [17:25:12] let's use the alert retro meet link wfan [17:25:17] Ok [17:26:45] Hi AndyRussG, following on from the conversation yesterday I just tested disabling/enabling campaigns from Special:CentralNotice on mobile and it seems to work consistently. So not sure what was happening when we were at the offsite, maybe it was network issues [17:28:34] (03Merged) 10jenkins-bot: Update smashpig from v0.8.2 to v0.8.3 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/818185 (owner: 10Damilare Adedoyin) [17:39:07] (03PS1) 10Wfan: update smashpig and di for civi [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/818193 [17:50:15] (03PS1) 10Damilare Adedoyin: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/818195 [17:56:48] (03PS2) 10Damilare Adedoyin: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/818195 [18:00:12] (03CR) 10Damilare Adedoyin: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/818195 (owner: 10Damilare Adedoyin) [18:00:59] 10fundraising-tech-ops: Fundraising access request for lmedley - https://phabricator.wikimedia.org/T314075 (10Dwisehaupt) [18:01:14] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/818195 (owner: 10Damilare Adedoyin) [18:01:54] 10fundraising-tech-ops: Fundraising access request for lmedley - https://phabricator.wikimedia.org/T314075 (10Dwisehaupt) Access approved: ` Date: Thu, 28 Jul 2022 10:01:58 From: Lisa Seitz Gruwell To: Megan Hernandez Cc: Dallas Wisehaupt, Laurice Medley, Greg Grossmeier Subject: Re: Onboarding access for L... [18:06:39] !log SmashPig updated from ffe5066d to 8e8f0017 [18:06:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:10:09] 10fundraising-tech-ops: Fundraising access request for lmedley - https://phabricator.wikimedia.org/T314075 (10Dwisehaupt) [18:34:55] (03Abandoned) 10Wfan: update smashpig and di for civi [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/818193 (owner: 10Wfan) [18:59:01] (03PS1) 10Wfan: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/818206 [19:01:03] (03CR) 10Wfan: [V: 03+2 C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/818206 (owner: 10Wfan) [19:08:30] !log civicrm upgraded from 3143dda9 to 497bddf7 [19:08:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:21:10] nice job wfan :) [19:51:15] Oh haha, team work ~ [20:05:29] šŸŽ‰ [20:08:00] (03CR) 10Damilare Adedoyin: "Looks good, just trying to get civi setup to properly test. However I added a small comment below." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809644 (https://phabricator.wikimedia.org/T310758) (owner: 10Wfan) [20:08:23] So civi setup is still running, Iā€™d get back to reviewing your patch in the morning wFan [20:09:27] Bye for now [20:10:01] Good night :) [20:55:33] 10fundraising-tech-ops: Fundraising access request for lmedley - https://phabricator.wikimedia.org/T314075 (10Dwisehaupt) [20:58:36] 10fundraising-tech-ops: Fundraising access request for lmedley - https://phabricator.wikimedia.org/T314075 (10Dwisehaupt) Created ssl client cert and sent it via email. Password sent via SMS. Civi account created and set with random password. Superset account created and set with random password. Superset passwo... [21:20:38] 10Fundraising Sprint Machine Unlearning, 10Fundraising-Backlog: 3DS needed on CarteBaincaire in France - https://phabricator.wikimedia.org/T313528 (10spatton) p:05Triageā†’03Unbreak! I know @EMartin, @TSkaff, @greg, @jgleeson and others are looking at this super actively; I am triaging it as Unbreak Now beca... [21:24:39] 10Fundraising Sprint Machine Unlearning, 10Fundraising-Backlog: 3DS needed on CarteBaincaire in France - https://phabricator.wikimedia.org/T313528 (10EMartin) @spatton Sam, Adyen is saying our declines are legitimate 3DS declines and are wanting to change some settings on our account to see if that improves th... [21:29:12] 10Fundraising Sprint Machine Unlearning, 10Fundraising-Backlog: 3DS needed on CarteBaincaire in France - https://phabricator.wikimedia.org/T313528 (10spatton) Thanks for that info @EMartin! [21:59:12] 10Fundraising-Backlog, 10fr-donorservices: Adyen donors think they are recurring July 2022 - https://phabricator.wikimedia.org/T313854 (10EMartin) From Adyen 28 July: //Hi Evelyn, I think you need to add the "storeDetails": "Save for my next payment", field here: https://docs.adyen.com/online-paymen... [22:14:21] 10Fundraising-Backlog, 10fr-donorservices: Adyen donors think they are recurring July 2022 - https://phabricator.wikimedia.org/T313854 (10EMartin) From Adyen 28 July: //Hi Evelyn, I think you need to add the "storeDetails": "Save for my next payment", field here: https://docs.adyen.com/online-paymen... [22:15:28] 10Fundraising-Backlog, 10fr-donorservices: Adyen donors think they are recurring July 2022 - https://phabricator.wikimedia.org/T313854 (10EMartin) From Adyen 28 July: //Hi Evelyn, I think you need to add the "storeDetails": "Save for my next payment", field here: https://docs.adyen.com/online-paymen... [22:18:12] 10Fundraising Sprint Machine Unlearning, 10Fundraising-Backlog: 3DS needed on CarteBaincaire in France - https://phabricator.wikimedia.org/T313528 (10EMartin) @Adyen is changing the settings on our account (on 28 July), to retry declined 3DS transactions without 3DS to see if we get a lift in approval rates on... [23:17:47] (03PS1) 10Wfan: Regerate recurring token if null for audit and test [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/818242 (https://phabricator.wikimedia.org/T307602) [23:18:32] (03Abandoned) 10Wfan: wip: Regerate recurring token if null for audit [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/790415 (https://phabricator.wikimedia.org/T307602) (owner: 10Wfan) [23:18:40] (03PS2) 10Wfan: Regerate recurring token if null for audit and test [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/818242 (https://phabricator.wikimedia.org/T307602) [23:19:26] (03Abandoned) 10Wfan: wip: Regerate recurring token if null for audit [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/813995 (https://phabricator.wikimedia.org/T307602) (owner: 10Wfan) [23:36:24] (03PS4) 10Wfan: WIP: need check if transaction number there before send to queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/790417 (https://phabricator.wikimedia.org/T307602) [23:39:36] 10fundraising-tech-ops: Fundraising access request for lmedley - https://phabricator.wikimedia.org/T314075 (10Dwisehaupt) [23:50:12] (03CR) 10CI reject: [V: 04-1] WIP: need check if transaction number there before send to queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/790417 (https://phabricator.wikimedia.org/T307602) (owner: 10Wfan) [23:51:18] PROBLEM - check_puppetrun on frauth1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): User[lmedley] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth1002&service=check_puppetrun [23:52:12] PROBLEM - check_puppetrun on frdev1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): User[lmedley] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_puppetrun [23:53:13] PROBLEM - check_puppetrun on frbast2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): User[lmedley] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2001&service=check_puppetrun [23:55:18] PROBLEM - check_puppetrun on frdev1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): User[lmedley] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_puppetrun [23:56:12] PROBLEM - check_puppetrun on frbast1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): User[lmedley] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast1001&service=check_puppetrun [23:56:18] PROBLEM - check_puppetrun on fran1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): User[lmedley] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_puppetrun [23:56:18] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): User[lmedley] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth1001&service=check_puppetrun [23:56:20] RECOVERY - check_puppetrun on frauth1002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth1002&service=check_puppetrun [23:57:12] RECOVERY - check_puppetrun on frdev1002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_puppetrun [23:58:12] RECOVERY - check_puppetrun on frbast2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2001&service=check_puppetrun