[05:24:11] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_35) - 10https://gerrit.wikimedia.org/r/827064 (owner: 10L10n-bot) [06:23:12] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10fr-donorservices: Donation Form Currency Issue PayPal USD/CNY - https://phabricator.wikimedia.org/T315592 (10krobinson) Thanks @Ejegg. Currently if donors select PayPal on the [[ https://donate.wikimedia.org/w/index.php?title=Spec... [06:33:53] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Donations appearing in Civi but not Adyen - https://phabricator.wikimedia.org/T315487 (10krobinson) Thanks for the update @jgleeson When you mention those 5.7k donors, are they all NL and iDEAL or is this a general Civi-Adyen thing... [07:20:49] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/827272 (owner: 10L10n-bot) [07:48:41] (03CR) 10Raimond Spekking: [C: 03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/827272 (owner: 10L10n-bot) [07:58:10] (03CR) 10Wangombe: [C: 03+1] "This seems ready for merge. Is there something else we may be waiting on?" [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/811580 (https://phabricator.wikimedia.org/T312007) (owner: 10Abijeet Patro) [10:19:35] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10fr-donorservices: Donation Form Currency Issue PayPal USD/CNY - https://phabricator.wikimedia.org/T315592 (10Pcoombe) I've removed PayPal from the China donatewiki form - [diff](https://donate.wikimedia.org/w/index.php?title=Templat... [10:31:41] (03CR) 10Nikerabbit: [C: 03+2] Remove usage of Translate RevTag class (031 comment) [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/811580 (https://phabricator.wikimedia.org/T312007) (owner: 10Abijeet Patro) [10:50:08] (03Merged) 10jenkins-bot: Remove usage of Translate RevTag class [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/811580 (https://phabricator.wikimedia.org/T312007) (owner: 10Abijeet Patro) [13:28:13] 10fundraising-tech-ops, 10Infrastructure-Foundations, 10SRE, 10netops: Upgrade pfw to Junos 20+ - https://phabricator.wikimedia.org/T295691 (10ayounsi) [13:41:08] 10fundraising-tech-ops, 10Infrastructure-Foundations, 10netops: Upgrade fasw to Junos 21 - https://phabricator.wikimedia.org/T316542 (10ayounsi) p:05Triage→03Low [13:41:30] 10fundraising-tech-ops, 10Infrastructure-Foundations, 10netops: Upgrade fasw to Junos 21 - https://phabricator.wikimedia.org/T316542 (10ayounsi) [14:03:51] Hi fr-tech I am back in New York [15:09:51] Hi Christine~ [15:14:41] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10FR-Braintree-Integration: Paypal audit parser should drop Braintree-initiated donations - https://phabricator.wikimedia.org/T315258 (10AnnWF) According to braintree's email: "Currently, there is no method or explicit parameter that... [15:20:47] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10FR-form-chooser-refactor: Document new gateway chooser logic and settings - https://phabricator.wikimedia.org/T315011 (10AnnWF) a:03AnnWF [15:49:59] (03PS1) 10Wfan: Opt out braintree transaction from paypal audit [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/827523 (https://phabricator.wikimedia.org/T315258) [15:51:59] (03CR) 10CI reject: [V: 04-1] Opt out braintree transaction from paypal audit [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/827523 (https://phabricator.wikimedia.org/T315258) (owner: 10Wfan) [16:00:06] (03PS2) 10Wfan: Opt out braintree transaction from paypal audit [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/827523 (https://phabricator.wikimedia.org/T315258) [16:01:04] 10Fundraising-Backlog, 10fundraising-tech-ops: Need new SSL cert & access for new laptop - https://phabricator.wikimedia.org/T316361 (10TSkaff) Ah makes sense, thanks @Dwisehaupt, Cold I trouble you to send me the SSL cert info? I can't seem to find it. [16:02:01] (03CR) 10CI reject: [V: 04-1] Opt out braintree transaction from paypal audit [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/827523 (https://phabricator.wikimedia.org/T315258) (owner: 10Wfan) [16:07:35] (03PS3) 10Wfan: Opt out braintree transaction from paypal audit [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/827523 (https://phabricator.wikimedia.org/T315258) [16:08:37] (03CR) 10CI reject: [V: 04-1] Opt out braintree transaction from paypal audit [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/827523 (https://phabricator.wikimedia.org/T315258) (owner: 10Wfan) [16:11:33] (03PS4) 10Wfan: Opt out braintree transaction from paypal audit [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/827523 (https://phabricator.wikimedia.org/T315258) [16:13:14] (03CR) 10CI reject: [V: 04-1] Opt out braintree transaction from paypal audit [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/827523 (https://phabricator.wikimedia.org/T315258) (owner: 10Wfan) [16:17:07] (03PS5) 10Wfan: Opt out braintree transaction from paypal audit [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/827523 (https://phabricator.wikimedia.org/T315258) [16:18:47] (03CR) 10CI reject: [V: 04-1] Opt out braintree transaction from paypal audit [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/827523 (https://phabricator.wikimedia.org/T315258) (owner: 10Wfan) [16:20:03] (03PS6) 10Wfan: Opt out braintree transaction from paypal audit [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/827523 (https://phabricator.wikimedia.org/T315258) [16:20:17] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Donations appearing in Civi but not Adyen - https://phabricator.wikimedia.org/T315487 (10Cstone) >>! In T315487#8179820, @jgleeson wrote: > I looked into the two transactions provided in the description, and I can see that Adyen sent... [16:32:38] hi fr-tech! [16:33:11] hellloo cstone wfan ejegg fr-tech :) [16:33:45] back in Mexico City now..... [16:33:50] hi AndyRussG [16:33:54] hope it was a good flight [16:34:01] hi AndyRussG ejegg I also have left Canada :P [16:34:15] how was the wedding cstone? [16:34:16] exodus! [16:34:45] hah it was a lot of work and chaos but it went well [16:34:47] wow, have a safe trip way back~ [16:34:56] cstone: photos? haha [16:35:16] ah cool cstone :) [16:36:08] hah i dont have many photos from day of as I was doing all the driving and helping [16:36:33] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10fr-donorservices: Donation Form Currency Issue PayPal USD/CNY - https://phabricator.wikimedia.org/T315592 (10EMartin) @krobinson. We don't have multi-curency setup yet with Paypal through Braintree yet. This was enabled in error an... [16:40:20] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10fr-donorservices: Donation Form Currency Issue PayPal USD/CNY - https://phabricator.wikimedia.org/T315592 (10Ejegg) @krobinson we CAN add China to the list of supported countries for PayPal, though. I guess fr-tech was under the imp... [16:49:22] 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising Sprint Machine Unlearning, 10Fundraising Sprint NaN is a Number, and 4 others: Create IPN listener for Braintree - https://phabricator.wikimedia.org/T303451 (10XenoRyet) [16:57:23] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10fr-donorservices: Donation Form Currency Issue PayPal USD/CNY - https://phabricator.wikimedia.org/T315592 (10EMartin) @Ejegg No objection to enable CN for PP express form. [17:10:47] 10Fundraising-Backlog, 10fundraising-tech-ops: Need new SSL cert & access for new laptop - https://phabricator.wikimedia.org/T316361 (10Dwisehaupt) Sure thing. I have resent the ssl cert via email and the password via SMS. [17:21:26] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Donations appearing in Civi but not Adyen - https://phabricator.wikimedia.org/T315487 (10XenoRyet) p:05Triage→03Unbreak! [17:27:33] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Donations appearing in Civi but not Adyen - https://phabricator.wikimedia.org/T315487 (10HNordeenWMF) launch of NL Banners has been delayed 1 day to allow time for this! Now scheduled for August 31 at 16 UTC [18:05:07] fr-tech so do we need to make iDEAL non-recurring work like recurring? https://wikitech.wikimedia.org/wiki/Fundraising/Data_and_flow/PSP_integrations/Adyen_Checkout#iDEAL_Recurring [18:06:37] AndyRussG: yes, I think something like that [18:06:48] We are already getting an AUTHORIZATION IPN from them [18:06:52] we originally decided to ignore the OFFER_CLOSED ipns should we not then? [18:07:02] and sending a PaymentCaptureJob to the queue [18:07:07] ohhh hmm [18:07:26] cstone I think we should just stop treating iDEAL donations in pending state as completed on the front-end [18:07:51] I'm just confirming that the donors will still get the TY letters based on the IPN that comes in when it succeeds [18:08:42] how do the TYs work for that recurring flow? [18:08:46] do they all go to pending or just some that get stuf? [18:08:48] stuf = stucl [18:08:50] omg [18:08:51] stuck [18:09:16] AndyRussG: TYs work the same for all the donations - we send them a few minutes after the civicrm_contribution row is created [18:10:22] AndyRussG: ejegg we'll need to communicate that the TYs/donations in civi will be delayed on waiting for the IPN because this used to confuse people when testing old adyen [18:10:23] cstone: good question, I think we send details to the pending table for all the iDEAL donations, and that the RecordPaymentCaptureJob does look at that DB [18:10:41] cstone let's look at how much of a delay that is... [18:10:56] just i remember people being confused its not instant [18:11:13] seems like it's only a minute or so [18:11:27] ah okay nice i guess i was thinking they were all like 20 minutse [18:12:29] I see 127960900.1 in the payments-wiki logs at 1:50:40, then the auth IPN comes in at 01:51:04 and the recordCaptureJob is 01:53:02 [18:12:38] nice [18:12:59] and donation QC at 01:53:16 [18:13:55] OK, so it's probably in the SmashPig class that we're mapping pending status to completed for iDEAL, right? [18:16:51] i see authorised = complete is that it? [18:17:02] cstone: in the IPN ? [18:17:10] oh ipns ok no [18:17:46] I think the IPN code is OK [18:17:54] We just need to change this line: https://phabricator.wikimedia.org/diffusion/WFSP/browse/master/PaymentProviders/Adyen/DirectDebitPaymentProvider.php$37 [18:18:08] ahh okay [18:39:14] 10fundraising-tech-ops, 10Patch-For-Review: Install and configure new host frdb1006 - https://phabricator.wikimedia.org/T312584 (10Jgreen) 05Open→03Resolved [18:39:37] hmm, so do we want to change the getPaymentDetailsSuccessfulStatuses as well? https://phabricator.wikimedia.org/diffusion/WFSP/browse/master/PaymentProviders/Adyen/DirectDebitPaymentProvider.php$53 [18:40:33] Also, I feel like the PENDING_POKE doesdn't actually happen for iDEAL [18:42:30] If iDEAEL dose not use the PENDING_POKE then we just needs to remove pending from both line? [18:42:51] I'm not totally sure, we should confirm [18:47:12] ejegg are you using our cloud to test the ipns or does anyone have that spreadsheet link [18:47:33] i know theres a link to it somewher ebut failing at finding it [18:48:11] I see jack and christine both mentioned OFFER_CLOSED, is that related? [18:49:10] its the other end of the issue I think wfan [18:49:26] we get that when the donor never completes the transaction through their bank [18:50:37] 10Fundraising-Backlog, 10fr-donorservices: Donation Form Error Message - https://phabricator.wikimedia.org/T316578 (10AMJohnson) [18:50:53] ok figured out the ipn thing just console being confusing [18:51:59] 10Fundraising-Backlog, 10fr-donorservices: Donation Form Error Message - https://phabricator.wikimedia.org/T316578 (10AMJohnson) [19:04:01] I updated instructions here if anyone else hasn't done it in a while https://wikitech.wikimedia.org/wiki/Fundraising/Development_tools/Testing#Instant_Payment_Notifications(IPN)_Messages_-_using_wmcloud [19:05:09] cstone: oh sorry, I am not actually testing anything yet - I got sidetracked doing some urgent Colombia taxes then helping with a poopy toddler [19:05:12] now back on task [19:05:19] no worries ejegg it was user error [19:14:55] 10Fundraising-Backlog, 10fr-donorservices: Donation Form Error Message - Country Detection - https://phabricator.wikimedia.org/T316578 (10AMJohnson) [19:19:59] 10Fundraising-Backlog, 10fr-donorservices: Donation Form Error Message - Country Detection - https://phabricator.wikimedia.org/T316578 (10Pcoombe) @AMJohnson Which email(s) could you reproduce this on? I tried a couple of recent seeds and they seemed to work okay. If you could share the link which gives the pr... [19:20:28] 10Fundraising-Backlog, 10donate.wikimedia.org, 10fr-donorservices, 10FR-Email: Donation Form Error Message - Country Detection - https://phabricator.wikimedia.org/T316578 (10Pcoombe) [19:23:20] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog: Review steps and create subtasks for Adyen Pending Transaction Resolver - https://phabricator.wikimedia.org/T316582 (10AndyRussG) [19:25:05] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog: Complete Adyen Checkout documentation to include donation statuses for all stages/permutations - https://phabricator.wikimedia.org/T316583 (10AndyRussG) [19:25:39] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog: Complete Adyen Checkout documentation to include donation statuses for all stages/permutations - https://phabricator.wikimedia.org/T316583 (10AndyRussG) [19:25:41] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog: Review steps and create subtasks for Adyen Pending Transaction Resolver - https://phabricator.wikimedia.org/T316582 (10AndyRussG) [19:25:43] 10Fundraising Sprint NaN is a Number, 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog: Adyen - auto settle stopped prior donors (Pending transaction Resolver) - https://phabricator.wikimedia.org/T299692 (10AndyRussG) [19:26:36] 10Fundraising-Backlog: Adyen - UML sequence diagrams - https://phabricator.wikimedia.org/T316584 (10AndyRussG) [19:27:20] 10Fundraising Sprint NaN is a Number, 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10Epic: Adyen - auto settle stopped prior donors (Pending transaction Resolver) - https://phabricator.wikimedia.org/T299692 (10AndyRussG) [19:27:49] 10Fundraising-Backlog, 10donate.wikimedia.org, 10fr-donorservices, 10FR-Email: Donation Form Error Message - Country Detection - https://phabricator.wikimedia.org/T316578 (10AMJohnson) I'm also receiving this error message when visiting donate.wikimedia.org or if you click donate from sidebar on WP. But no... [19:28:34] 10fundraising-tech-ops: install/configure frpm1002 - https://phabricator.wikimedia.org/T299069 (10Jgreen) 05Resolved→03Open [19:31:10] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog: Document how to test Adyen Pending Transaction Resolver locally - https://phabricator.wikimedia.org/T316585 (10AndyRussG) [19:31:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: slave query error: Access denied: you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [19:32:13] cstone: ok, now I'm thinking we just want to change the DonationInterface code [19:32:20] 10Fundraising-Backlog, 10fr-donorservices: Adyen duplicate donations after donors see error message - August 2022 - https://phabricator.wikimedia.org/T316157 (10XenoRyet) p:05Triage→03High [19:32:26] https://phabricator.wikimedia.org/diffusion/EDOI/browse/master/adyen_gateway/adyen_checkout.adapter.php$405 [19:33:30] since the pending status really should be considered a 'success' for the purposes of showing the donor the TY page [19:35:37] 10Fundraising-Backlog: Write code for Adyen Pending Transaction Resolver - https://phabricator.wikimedia.org/T316587 (10AndyRussG) [19:36:16] PROBLEM - check_mysql on frdb1006 is CRITICAL: slave query error: Access denied: you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [19:37:12] 10Fundraising-Backlog: Unit tests for Adyen Pending Transaction Resolver - https://phabricator.wikimedia.org/T316588 (10AndyRussG) [19:39:06] 10Fundraising-Backlog: Document how to test Adyen Pending Transaction Resolver locally - https://phabricator.wikimedia.org/T316585 (10AndyRussG) [19:39:27] 10Fundraising-Backlog, 10donate.wikimedia.org, 10fr-donorservices, 10FR-Email: Donation Form Error Message - Country Detection - https://phabricator.wikimedia.org/T316578 (10Pcoombe) p:05Triage→03Unbreak! Confirming, if I visit donate.wikimedia.org now it always redirects me to the unknown country http... [19:40:53] fr-tech: ^ this is pretty serious. Not sure what's going on, but redirection to the correct country on donatewiki isn't working [19:41:12] PROBLEM - check_mysql on frdb1006 is CRITICAL: slave query error: Access denied: you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [19:41:39] pcoombe: we're just talking about it in the daily standup [19:41:57] it would be the same component geolocating people on the main wikis for the banners [19:42:30] (varnish adds the cookie) [19:42:32] pcoombe: https://meet.google.com/mkm-dptq-dqq?authuser=0 if you want to join us [19:42:33] I still have a GeoIP cookie that looks right though [19:43:18] oh weird [19:43:42] pcoombe: do you have a moment to join us to investigate? [19:43:59] 10Fundraising-Backlog, 10donate.wikimedia.org, 10fr-donorservices, 10FR-Email: Donation Form Error Message - Country Detection - https://phabricator.wikimedia.org/T316578 (10Pcoombe) Huh, the reason I couldn't reproduce this earlier is because it seems to work when logged in to donatewiki. But not when log... [19:44:52] yeah gimme a sec. Just posted on the phab, weirdly it seems to work when logged in to donatewiki (not that that's much use to most of the world!) [19:46:12] PROBLEM - check_mysql on frdb1006 is CRITICAL: slave query error: Access denied: you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [19:51:16] PROBLEM - check_mysql on frdb1006 is CRITICAL: slave query error: Access denied: you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [19:56:16] PROBLEM - check_mysql on frdb1006 is CRITICAL: slave query error: Access denied: you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [19:57:08] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Adyen iDeal code seems to go through DirectDebitPaymentProvider - https://phabricator.wikimedia.org/T316368 (10XenoRyet) [19:57:10] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Amazon requesting URL removal to avoid violation - https://phabricator.wikimedia.org/T316389 (10XenoRyet) [19:57:12] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Alerts, 10MW-1.39-notes (1.39.0-wmf.15; 2022-06-06): SmashPig::queue2civicrm find a completed dlocal payment mark as failed and removed from queue table - https://phabricator.wikimedia.org/T309799 (10XenoRyet) [19:57:14] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Merge and go to listing redirect fail - https://phabricator.wikimedia.org/T309801 (10XenoRyet) [19:57:16] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Adyen duplicate donations after donors see error message - August 2022 - https://phabricator.wikimedia.org/T316157 (10XenoRyet) [19:57:19] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10donate.wikimedia.org, 10fr-donorservices, 10FR-Email: Donation Form Error Message - Country Detection - https://phabricator.wikimedia.org/T316578 (10XenoRyet) [20:01:16] PROBLEM - check_mysql on frdb1006 is CRITICAL: slave query error: Access denied: you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [20:06:09] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10donate.wikimedia.org, 10fr-donorservices, 10FR-Email: Donation Form Error Message - Country Detection - https://phabricator.wikimedia.org/T316578 (10Dzahn) https://gerrit.wikimedia.org/r/827566 is being reverted as we speak [20:06:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: slave query error: Access denied: you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [20:09:10] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10donate.wikimedia.org, 10fr-donorservices, 10FR-Email: Donation Form Error Message - Country Detection - https://phabricator.wikimedia.org/T316578 (10greg) Thanks to SRE for engaging with our "did something happen with geoip cook... [20:09:31] ACKNOWLEDGEMENT - check_mysql on frdb1006 is CRITICAL: slave query error: Access denied: you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation Dwisehaupt known - fixing. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [20:11:19] cstone: anyway, does it sound good to you to just change this line in DonationInterface, leaving the SmashPig code identical? https://phabricator.wikimedia.org/diffusion/EDOI/browse/master/adyen_gateway/adyen_checkout.adapter.php$405 [20:11:36] so skip the donations queue push for all ideal, not just recurring [20:12:19] and yes let's test that the iDEAL Authorization success IPN really does end up getting to the donations queue, too [20:16:08] can someone else confirm donatewiki working properly again? [20:16:43] fr-tech greg-g works for me now too! [20:16:43] works for me! \o/ [20:16:44] working for me now greg-g [20:16:49] awesome [20:18:42] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10donate.wikimedia.org, 10fr-donorservices, 10FR-Email: Donation Form Error Message - Country Detection - https://phabricator.wikimedia.org/T316578 (10greg) 05Open→03Resolved Alright, that revert did the trick. Thanks again to... [20:20:08] thanks all! I'm going to log off now. have a good rest of the day/night! [20:20:18] toodles [20:21:35] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10donate.wikimedia.org, 10fr-donorservices, 10FR-Email: Donation Form Error Message - Country Detection - https://phabricator.wikimedia.org/T316578 (10AMJohnson) Thank you all for the quick fix! Inviting donors to retry. [20:23:10] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10donate.wikimedia.org, 10fr-donorservices, 10FR-Email: Donation Form Error Message - Country Detection - https://phabricator.wikimedia.org/T316578 (10greg) @AMJohnson great! Just make sure they don't accidentally try to reload us... [20:26:53] sure ejegg just need to make sure it doesn't get stuck in any recurring only code on the smashpig side too [20:36:16] RECOVERY - check_mysql on frdb1006 is OK: Uptime: 1138190 Threads: 6 Questions: 60827227 Slow queries: 4 Opens: 1087 Open tables: 896 Queries per second avg: 53.442 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [20:38:14] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10donate.wikimedia.org, 10fr-donorservices, 10FR-Email: Donation Form Error Message - Country Detection - https://phabricator.wikimedia.org/T316578 (10AMJohnson) Good Point @greg! When we invite to retry we point them to our stand... [21:04:17] cstone sorry, I'm not sure I understand the iDEAL problem as well as I thought - looks like the code already correctly handles the 'Pending' status coming back from Adyen [21:04:31] so I guess these donors are coming back to us with status 'Authorized' ? [21:04:40] hmm [21:04:56] why would that not get to adyens side at all though [21:07:28] 10Fundraising-Backlog, 10fr-donorservices: Adyen iDEAL TY email delay - https://phabricator.wikimedia.org/T316602 (10MBeat33) [21:07:46] 10Fundraising-Backlog, 10fr-donorservices: Adyen iDEAL TY email delay - https://phabricator.wikimedia.org/T316602 (10MBeat33) p:05Triage→03Medium [21:14:13] unsure [21:14:24] oh, but we should be able to see that in the logs at least [21:23:32] yeah, hmm, they do seem to be coming in as Authorized [21:27:19] shoot, and they're getting the Authorized IPNs as well [21:27:51] oh wait, this one really is in Adyen [21:28:00] even though we got an OFFER_CLOSED [21:28:30] ok, I need to look at the logs for the examples in the ticket I guess, not just random OFFER_CLOSED IPNs from today [21:31:18] darn, we're not logging the raw response from the get details lookup [21:31:49] but by the time it gets to the payments_init log line it is complete:process [21:32:57] locally if we get the status pending I get pending:process in the payments_init log line [21:33:24] OK, I guess maybe we start logging that raw response? [21:34:35] anyway, for the ones mentioned in the ticket they DO NOT get the Authorized IPN [21:34:51] so the one-line DonationInterface fix would work [21:35:09] I'mma look for more offer_closed from today [21:38:59] ok, 127979870.1 fits the bill - pushed to donations but not in the console [21:42:52] (03PS1) 10Ejegg: Adyen: log raw details response [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/827593 (https://phabricator.wikimedia.org/T315487) [21:43:01] cstone: would you mind reviewing that? [21:43:24] I'd like to at least get one of those raw responses logged before we push out a behavioral change [21:43:47] and it looks like we're still getting enough iDEAL today that maybe we can catch one [21:44:01] lognitive behavioral therapy [21:44:12] yep yep [21:45:04] (03CR) 10CI reject: [V: 04-1] Adyen: log raw details response [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/827593 (https://phabricator.wikimedia.org/T315487) (owner: 10Ejegg) [21:45:08] haha [21:45:49] arr, we already json_decode the rawResponse [21:45:59] not quite as raw as all that, then [21:46:38] sorry had to relocate due to thunderstorm band but im back now [21:47:02] (03PS2) 10Ejegg: Adyen: log raw details response [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/827593 (https://phabricator.wikimedia.org/T315487) [21:49:15] Squeaky Pete and the Thunderstorm Band? [21:49:16] when i was in vancouver it thundered and lightninged one time and it was the talk of the next three days [21:50:33] not like Lake Maracaibo then? https://en.wikipedia.org/wiki/Catatumbo_lightning [21:50:54] "It originates from a mass of storm clouds at an altitude of more than 1 km (0.6 mi), and occurs during 140 to 160 nights a year, nine hours per day, and with lightning flashes from 16 to 40 times per minute." [21:51:04] ejegg should we only log for the ideal one? or general ? [21:51:09] nah just a normal whatever this is? haha https://on.windy.com/6u5h0 [21:51:37] wfan I think it's OK to log for the general case as long as it's at debug level [21:52:13] ooh, that's a fun blob [21:52:23] OK [21:53:14] (03CR) 10Wfan: [C: 03+2] "Looks good, hope we can find the root cause for ideal status :)" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/827593 (https://phabricator.wikimedia.org/T315487) (owner: 10Ejegg) [21:53:15] cstone wow there are lots of long straight streaks on there if you zoom out / look around. Contrails? [21:53:20] thanks wfan ! [21:53:36] oh thats just radar being radar [21:53:42] ooh cool realtime lightning strikes [21:53:44] interference from something [21:53:50] interesting [21:54:09] so other weather sites filter that out [21:54:48] ive seen it on nws radar too [21:55:54] (03Merged) 10jenkins-bot: Adyen: log raw details response [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/827593 (https://phabricator.wikimedia.org/T315487) (owner: 10Ejegg) [21:56:37] oh hmm, damilare|away has a patch in the deploy queue [21:56:43] think I'll just cherry-pick the logging for now [21:57:07] (03PS1) 10Ejegg: Adyen: log raw details response [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/827595 (https://phabricator.wikimedia.org/T315487) [21:57:12] (03CR) 10Ejegg: [C: 03+2] Adyen: log raw details response [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/827595 (https://phabricator.wikimedia.org/T315487) (owner: 10Ejegg) [21:58:29] (03Merged) 10jenkins-bot: Adyen: log raw details response [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/827595 (https://phabricator.wikimedia.org/T315487) (owner: 10Ejegg) [21:58:36] (03PS1) 10Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/827596 [21:58:38] (03CR) 10Ejegg: [C: 03+2] Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/827596 (owner: 10Ejegg) [22:08:03] (03Merged) 10jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/827596 (owner: 10Ejegg) [22:08:19] 10Fundraising-Backlog, 10fundraising-tech-ops, 10SRE, 10Traffic-Icebox, and 2 others: SSL cert for links.email.wikimedia.org - https://phabricator.wikimedia.org/T188561 (10greg) [22:16:55] !log payments-wiki upgraded from a63b300e to dc6d899d [22:16:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:40:09] oh shoot, are we not logging at debug level in DonationInterface? I was fooled by all those payments.debug lines from the SmashPig layer [22:40:26] somehow I thought we made that pass through [22:42:31] ok, turned that on [22:42:49] and noticed we still have debug logging on for PayPal Express Checkout and Amazon as well [22:44:15] hmmm cool [22:44:33] quiddity: 3 [22:44:50] oops sorry quiddity mis-typed :) also hii :) [22:44:55] * quiddity denies everything [22:45:09] ;) [22:47:34] huh, adyen's debug-level logging goes to payments-adyen but paypal EC's debug-level logging goes to payments.debug [22:48:07] ah, maybe the tag filtering? [22:48:28] adyen is using the predictable adyen_gateway while paypal is tagged paypal_ec_gateway [22:49:19] got some iDEAL getStatus details logged now, but it's an actually-completed one [22:51:36] ok, will try to figure out API call chaining in Civi while I wait for that to log more [23:32:59] 10fundraising-tech-ops, 10Infrastructure-Foundations, 10SRE, 10netops: Upgrade fasw to Junos 21 - https://phabricator.wikimedia.org/T316542 (10Dwisehaupt) @ayounsi We have a maintenance week for frack scheduled for Sep 26-30. Would sometime that week be good for you? We could do fasw1-c-codfw before then i... [23:41:32] ejegg: I just remmebered that offers are in a different UI in adyens console so we can look them up [23:41:38] not sure if you already saw that though [23:45:51] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices, 10MW-1.39-notes (1.39.0-wmf.27; 2022-08-29): Donations appearing in Civi but not Adyen - https://phabricator.wikimedia.org/T315487 (10Cstone) You can see these in the Adyen console by searching under Transactions -> Offers. Not sur... [23:55:38] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices, 10MW-1.39-notes (1.39.0-wmf.27; 2022-08-29): Donations appearing in Civi but not Adyen - https://phabricator.wikimedia.org/T315487 (10Cstone) 128037526.1 is an example where they tried to donate twice, one was successful and one wa...