[01:01:35] ok, I'll deploy that now [01:03:46] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/817399 [01:03:49] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/817399 (owner: 10Ejegg) [01:04:32] ohhh thx ejegg! [01:04:59] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/817399 (owner: 10Ejegg) [01:05:31] (03PS1) 10Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/817400 [01:05:33] or should I do that? [01:05:36] (03CR) 10Ejegg: [C: 03+2] Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/817400 (owner: 10Ejegg) [01:05:48] AndyRussG: if you want to, be my guest! [01:06:00] Just as soon as that core patch merges [01:06:11] oki ejegg will do! [01:06:58] I'm not sure if wfan pushed out the library update she had merged to our branch of core, so there may be more than just that one merge going out. [01:08:00] I did not pushed out the library today yet [01:08:18] Update* not today [01:10:22] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Investigate queue consumer slowdown in pretest of June 13 2022 - https://phabricator.wikimedia.org/T313000 (10Ejegg) After seeing the slowdown, we enabled debug logging for 20 minutes to see if the additional log lines would help pinpoint the slow part of... [01:12:38] ok, thanks wfan - so AndyRussG you should expect this change to go out as well: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/817297 [01:14:09] oki [01:14:47] (03Merged) 10jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/817400 (owner: 10Ejegg) [01:14:53] (03CR) 10Ejegg: "Oops wfan, looks like d321b5ddcac35732fd06b07c4280609ac6c16cb1 is on the master branch of DonationInterface, not the deployment branch. Le" [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/817297 (owner: 10Wfan) [01:17:15] Yeah, this needs update, since di is not yet get updated, also the version v2.5.7.3 got some error as fatal: reference is not a tree: 55ca988e88aaf550738cbd08dd93c0284f6f940d, do you have time for a chat? [01:17:34] sure wfan, let me just find my headphones [01:17:41] thanks so much~ [01:18:27] https://meet.google.com/aio-mocx-ggt?authuser=0 [01:21:40] also since the 3ds got merged, should we create another tag? or this is not something we want to use in crm? [01:23:10] not used in CRM [01:26:14] ejegg: wfan: so I'll hold of on pushing out to prod I think? [01:26:39] I could walk the dog first... no rush from my perspective [01:26:49] AndyRussG: no, no need [01:27:03] I updated the submodule back to deployment in my latest core PS [01:29:16] ejegg: wfan: ok so this is the current tip of the mw core fundraising/REL1_35 branch https://gerrit.wikimedia.org/r/c/mediawiki/core/+/817400 [01:29:21] so that's what will go out [01:30:31] wfan: ejegg: did anyone smoke test the horrific shim's new code path for execution under Mediawiki? [01:30:51] or I should say, new code in the MW execution code path? [01:30:56] I ran the tests, but I haven't tried a test donation [01:31:28] will do that now [01:31:49] ok thx! I'll try running something that goes through that exact code [01:32:47] Just loading any payment form should do it [01:32:57] they all need to instantiate an adapter [01:33:07] and this code is run in the contstructor [01:33:43] ah yeah I see indeed [01:39:09] ok yeah all good [01:41:42] I really want to ctrl-C during that "Refreshing git metadata............" [01:41:46] just for old time's sake [01:41:50] haha, it traps it now! [01:41:55] I tried it by accident [01:42:42] maybe you can mess it up if you ctrl-c several times in quick succession [01:44:11] (03PS2) 10Wfan: Update SmashPig to 0.8.2 and DI to 2.5.7.3 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/816830 [01:44:47] !log update payments-wiki 4487bd31 -> 589bb64 [01:44:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:46:01] we should fix those client-side errors [01:46:42] (03CR) 10Ejegg: [C: 03+2] "Looks good!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/816830 (owner: 10Wfan) [01:47:04] and also those DBReadOnlyErrors [01:49:26] hmm dlocal here makes me click an extra "ok" ("aceptar") before I go to the t-y page [01:54:12] ok ejegg wfan donation went through fine on prod, no flood of error messages, got a nice ty message from Maryana [01:54:25] hooray! [01:54:27] Nice~ [01:54:29] :) [01:54:39] gonna take doggie on his daily tour of neighbourhood bushes and lamp posts [01:54:43] back in a bit! [01:54:46] enjoy [01:54:59] I'll tell him you said so ;p [02:00:32] (03Merged) 10jenkins-bot: Update SmashPig to 0.8.2 and DI to 2.5.7.3 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/816830 (owner: 10Wfan) [02:01:05] (03PS10) 10Wfan: Add test for civi run audit [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809644 (https://phabricator.wikimedia.org/T310758) [02:22:56] 10Fundraising-Backlog: 3DS needed on CarteBaincaire in France - https://phabricator.wikimedia.org/T313528 (10Ejegg) Hi @EMartin, we have just deployed that fix for the problem that @jgleeson found, removing the check on brand/card type. This should make sure browserInfo is sent for all EUR transactions in France... [02:23:27] 10Fundraising Sprint Machine Unlearning, 10Fundraising-Backlog: 3DS needed on CarteBaincaire in France - https://phabricator.wikimedia.org/T313528 (10Ejegg) [02:24:39] 10Fundraising Sprint Machine Unlearning, 10Fundraising-Backlog, 10MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), 10Patch-For-Review: Adyen Carte Bancaire 3D Secure Headers not being set - https://phabricator.wikimedia.org/T313756 (10Ejegg) Fix deployed! Thanks much @jgleeson and @AndyRussG . [02:25:36] 10Fundraising Sprint Machine Unlearning, 10Fundraising-Backlog, 10FR-Alerts, 10FR-Smashpig, 10MediaWiki-extensions-DonationInterface: Don't send failmail on Adyen unsupported card - https://phabricator.wikimedia.org/T313069 (10Ejegg) This fix went out with the Adyen 3DSecure fix. Hopefully we will never... [03:28:20] 10Fundraising Sprint Machine Unlearning, 10Fundraising-Backlog: 3DS needed on CarteBaincaire in France - https://phabricator.wikimedia.org/T313528 (10EMartin) @jgleeson @Ejegg Great news both! We will monitor France activity and ensure we are good to go with the campaign. Your attention here is much apprecia... [05:17:52] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_38) - 10https://gerrit.wikimedia.org/r/817483 (owner: 10L10n-bot) [05:27:13] PROBLEM - check_procs on frdb2003 is CRITICAL: PROCS CRITICAL: 2362 processes https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_procs [05:32:11] RECOVERY - check_procs on frdb2003 is OK: PROCS OK: 1327 processes https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_procs [07:37:15] PROBLEM - check_kafkatee on frban1001 is CRITICAL: CRITICAL: kafka-jumbo1008:down https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_kafkatee [07:42:15] PROBLEM - check_kafkatee on frban1001 is CRITICAL: CRITICAL: kafka-jumbo1009:down https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_kafkatee [07:47:15] RECOVERY - check_kafkatee on frban1001 is OK: OK: brokers:9 topics:1 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_kafkatee [07:59:02] 10Fundraising Sprint Machine Unlearning, 10Fundraising-Backlog, 10FR-Braintree-Integration: Update PayPal IPN listener to discard donations initiated via Braintree - https://phabricator.wikimedia.org/T313512 (10Damilare) a:03Damilare [08:52:46] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: This code snippet might be the way to get rid of `WMFTransaction` - https://phabricator.wikimedia.org/T313846 (10Eileenmcnaughton) clickbait for geeks.... [09:33:29] (03PS1) 10Damilare Adedoyin: Update PayPal IPN listener to discard donations initiated via Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/817733 (https://phabricator.wikimedia.org/T313512) [10:05:25] (03PS2) 10Damilare Adedoyin: Update PayPal IPN listener to discard donations initiated via Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/817733 (https://phabricator.wikimedia.org/T313512) [17:08:14] (03CR) 10Ejegg: [C: 03+2] "Looks good!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/817733 (https://phabricator.wikimedia.org/T313512) (owner: 10Damilare Adedoyin) [17:09:59] (03CR) 10Ejegg: [C: 04-1] "Sorry, just thought of a change or two to request!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/817733 (https://phabricator.wikimedia.org/T313512) (owner: 10Damilare Adedoyin) [17:12:54] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Investigate queue consumer slowdown in pretest of June 13 2022 - https://phabricator.wikimedia.org/T313000 (10greg) There's an en6C pre-test today so that would be a good time to do some investigation. [17:21:33] 10Fundraising-Backlog: Donation shows in Adyen but not in Civi - NL - https://phabricator.wikimedia.org/T313826 (10EMartin) @Ejegg Hi Elliott, we are confused as to why the donations we ran yesterday are not in Civi. Are you saying we shouldn't copy links? Is this being worked on? Or is this the state of thing... [17:23:06] 10Fundraising-Backlog: Donation shows in Adyen but not in Civi - NL - https://phabricator.wikimedia.org/T313826 (10EMartin) @Ejegg Also, if we refund this transaction VQL7RPCTHS7WTM42, will the refund also not be reflected in Civi? [17:25:27] 10Fundraising-Backlog: Donation shows in Adyen but not in Civi - NL - https://phabricator.wikimedia.org/T313826 (10greg) It's fine to copy links, but if copying *from Acoustic* be sure to replace the "%%isolang%%" bit with "en" or "es" or whatever language code makes sense. Context: Acoustic uses placeholders t... [17:26:34] 10Fundraising-Backlog: Donation shows in Adyen but not in Civi - NL - https://phabricator.wikimedia.org/T313826 (10Ejegg) @EMartin if you copy links directly from an email template, you need to edit them by hand so they will be valid. When Acoustic sends an email from a template, it replaces a lot of tokens with... [17:27:05] 10Fundraising-Backlog: Donation shows in Adyen but not in Civi - NL - https://phabricator.wikimedia.org/T313826 (10greg) (jinx) [17:27:24] 10Fundraising-Backlog: Donation shows in Adyen but not in Civi - NL - https://phabricator.wikimedia.org/T313826 (10Ejegg) @EMartin regarding the refund, if the parent transaction did not make it to Civi, neither will the refund. [17:28:33] Ok so why didn't the parent transaction make it to Civi in the example Rakhi posted (not pulled from acoustic or was it?) [17:31:46] Based on the logs (from elliot's comment at https://phabricator.wikimedia.org/T313826#8105569): because it was using an incomplete/broken url. If the url has "%%isolang%%" in it things will fail. No real user should be presented with such an url. [17:32:31] Ok, got it. Makes sense. Noted on the edits that need to be made. [17:33:43] +1 [17:34:01] 10Fundraising-Backlog: Donation shows in Adyen but not in Civi - NL - https://phabricator.wikimedia.org/T313826 (10EMartin) Noting this here for posterity. Ok so why didn't the parent transaction make it to Civi in the example Rakhi posted (not pulled from acoustic or was it?) 10:31 AM greg Based on th... [17:37:16] My next question is this: We have another example which came in through an email (not a copied link) L2J3WSWXGCCLHK42 and it also did not show up in Civi. This was stopped for fraud review and it was manually settled. [17:41:58] greg-g. In the theme of oddities on things not showing up in Civi, we logged a task this month https://phabricator.wikimedia.org/T313702 where a recurring donation does not get to Civi. What could be the root cause of these sorts of issues? [17:46:35] ok, for that task from 3 days ago, we briefly looked at it yesterday during our backlog review but it requires some more investigation. Given we are focused on braintree for the rest of this sprint I don't think we'll be able to do more investigation until next week. Unless this is a highpriority/needs to bump braintree request? [17:49:43] emartin: ^ (sorry forgot to ping you on my reply) [17:50:28] no, not a priority, just a mystery we'd like to solve for to avoid possible future problems. Thanks! [17:52:15] emartin: if there's more of them coming in please do add the references to the task and we'll take a look next week. Thanks! [18:35:56] (03PS1) 10Ejegg: Debug logging for queue speed investigation [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/817874 (https://phabricator.wikimedia.org/T313000) [18:43:14] 10Fundraising-Backlog, 10fr-donorservices: Donation form issues if back button utilized - https://phabricator.wikimedia.org/T313969 (10AMJohnson) [18:49:45] (03CR) 10Wfan: [C: 03+2] "Made a donation and run the ./consume-queues.sh, looks no syntax error." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/817874 (https://phabricator.wikimedia.org/T313000) (owner: 10Ejegg) [18:53:49] 10Fundraising-Backlog, 10FR-Adyen, 10FR-Japan, 10Epic: EPIC: Japanese form updates - https://phabricator.wikimedia.org/T299565 (10spatton) [18:53:54] 10Fundraising-Backlog, 10FR-Adyen, 10FR-Japan, 10Epic: EPIC: Japanese form updates - https://phabricator.wikimedia.org/T299565 (10spatton) [18:56:59] (03PS1) 10Umherirrender: schema: Convert Unique Index to Primary Key on cn_notice_languages [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/817879 (https://phabricator.wikimedia.org/T312270) [18:57:01] (03PS1) 10Umherirrender: schema: Convert Unique Index to Primary Key on cn_notice_projects [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/817880 (https://phabricator.wikimedia.org/T312272) [18:57:03] (03PS1) 10Umherirrender: schema: Convert Unique Index to Primary Key on cn_notice_countries [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/817881 (https://phabricator.wikimedia.org/T312273) [18:57:05] (03PS1) 10Umherirrender: schema: Convert Unique Index to Primary Key on cn_notice_regions [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/817882 (https://phabricator.wikimedia.org/T312274) [19:03:30] (03Merged) 10jenkins-bot: Debug logging for queue speed investigation [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/817874 (https://phabricator.wikimedia.org/T313000) (owner: 10Ejegg) [19:13:21] (03PS1) 10Ejegg: Debug logging for queue speed investigation [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/817888 (https://phabricator.wikimedia.org/T313000) [19:13:26] (03CR) 10Ejegg: [C: 03+2] Debug logging for queue speed investigation [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/817888 (https://phabricator.wikimedia.org/T313000) (owner: 10Ejegg) [19:16:46] !log updated Fundraising CiviCRM from b4a7154a to e0962be6 [19:16:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:16:55] ok, that's the extra logging out [19:19:37] hmm, I might have to add those extra -v s on to the drush command too [19:20:16] ooh, deadlocks [19:20:25] what's happening in the db? [19:21:20] ejegg: thanks for pushing that out. I don't see deadlocks? [19:21:29] yeah... that's a nasty dedupe query [19:21:47] jgleeson: in the p-c log donations_queue_consume-20220727-192002.log [19:22:24] huh. I don't have any failmail for today [19:22:44] oh wait you're just browsing it to confirm the logging stuff right? [19:22:58] deadlocks with no failmail! ? [19:23:01] yeah [19:23:13] I think we requeue when we can catch it correctly [19:23:45] user 818 [19:23:57] argh [19:24:17] khansen [19:24:36] so I'll kill it - it's been running for 6000+ seconds already [19:27:00] sounds good [19:27:12] back in a bit [19:28:49] ok, that should get all the debug logging in the file [19:28:56] I'mma head out for a bit too [19:29:05] should be back shortly after the start of the test [20:30:10] (03PS3) 10Damilare Adedoyin: Update PayPal IPN listener to discard donations initiated via Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/817733 (https://phabricator.wikimedia.org/T313512) [21:28:04] Dami, you code did not change haha [22:25:55] 10Fundraising-Backlog, 10fr-donorservices: Adyen donors think they are recurring July 2022 - https://phabricator.wikimedia.org/T313854 (10EMartin) @mbeattie. We'll have Adyen check this out on their end. Thanks for flagging. [22:30:47] 10Fundraising-Backlog, 10fr-donorservices: Adyen donors think they are recurring July 2022 - https://phabricator.wikimedia.org/T313854 (10Ejegg) @mbeattie @EMartin could this be because we turned on post-payment monthly convert? When we have that turned on, we always tokenize the card details with Adyen. If t... [22:32:49] hah ejegg you answered faster than me there [22:33:15] :) [22:33:29] I wonder if there's a different flow we could be using for Adyen [22:33:42] like to get a token from an earlier PSP reference? [22:34:27] or is that new update they released right after we finished ours different there [23:03:54] 10Fundraising-Backlog, 10fr-donorservices: Adyen donors think they are recurring July 2022 - https://phabricator.wikimedia.org/T313854 (10EMartin) @ejegg. I ran a transaction in Spain and Austria just now as one time and they showed up in the Adyen console as recurring. Both transactions had the post payment... [23:06:19] 10Fundraising-Backlog, 10fr-donorservices: Adyen donors think they are recurring July 2022 - https://phabricator.wikimedia.org/T313854 (10EMartin) @ejegg. Does this indicate a problem on the Adyen end, i.e., they are not discarding the token as we are or is there something on our end we are invoking that we sh...