[00:33:38] eileen: stepping through this locally do the html tags ever get looked at then? if they dont get added to the final text and I guess its just using the /n? [00:40:03] ooh i see whats happening ok ignore that [00:44:20] (03CR) 10Cstone: [C: 03+2] "

s are gone" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/926039 (https://phabricator.wikimedia.org/T337919) (owner: 10Eileen) [00:56:58] (03Merged) 10jenkins-bot: Fix for mis-rendered html in failure email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/926039 (https://phabricator.wikimedia.org/T337919) (owner: 10Eileen) [01:16:13] (03PS1) 10Cstone: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/926084 [01:16:43] eileen: ^ those other ones ok to also go out, mainly that transaction one [01:16:48] are those * [01:17:08] cstone: yes [01:17:12] thanks! [01:18:17] what did i miss that the tests run in 30 seconds?? [01:18:36] maybe these always did [01:18:39] my brain is failing me today [01:19:20] (03CR) 10Cstone: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/926084 (owner: 10Cstone) [01:20:13] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/926084 (owner: 10Cstone) [01:22:30] !log civicrm upgraded from 3819d6d1 to bcc8fccc [01:22:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:24:06] 10Fundraising-Backlog: Formatting issue in 'Renew Your Recurring' email sent from Civi - https://phabricator.wikimedia.org/T337919 (10Cstone) I just deployed the fix to this and the visible html tags are gone. [01:43:22] sorry for the absentee-ism folks. Mother-in-law leaves tomorrow and there have been some last minute things to prepare [01:44:17] cstone: i think on the merge to deploy it only does the linting, right? since we delete the tests [01:45:22] yeah ejegg brain fart there [01:45:31] i had literally just deleted 3 tests... [01:45:43] :) [01:46:09] hmm, so i need to see why there are so few rows in the unsubs file with that rebuild [01:46:48] oh and I need to deploy that column switch fix [01:46:53] let's do that before I forget [01:49:18] (03PS1) 10Ejegg: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/926121 [01:49:24] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/926121 (owner: 10Ejegg) [01:50:27] (03Merged) 10jenkins-bot: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/926121 (owner: 10Ejegg) [01:53:37] !log fundraising python tools upgraded from 759d4c89 to 2ca83336 [01:53:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:07:06] 10Fundraising-Backlog, 10fundraising-tech-ops: migrate Silverpop export job from civi1001/frdev1001 to civi1002+frdb1006 - https://phabricator.wikimedia.org/T337498 (10Ejegg) [02:07:55] 10Fundraising-Backlog, 10fundraising-tech-ops: migrate Silverpop export job from civi1001/frdev1001 to civi1002+frdb1006 - https://phabricator.wikimedia.org/T337498 (10Ejegg) I just switched the scheduled job config over so it should run from civi1002 instead of civi1001 in a little under an hour [02:36:16] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1471 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [02:37:20] ACKNOWLEDGEMENT - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1471 Elliott Eggleston rebuilding unsubscribes table https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [02:46:16] RECOVERY - check_mysql on frdb1006 is OK: Uptime: 50574 Threads: 6 Questions: 1931800 Slow queries: 74766 Opens: 2113 Open tables: 2100 Queries per second avg: 38.197 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [02:50:07] dang, I think I have to push that scheduled run back an hour [02:50:33] it does have a "# TODO: push this later once it's incremental" comment [03:15:32] (03CR) 10Ejegg: [C: 04-1] "This is really hard to review because of the changes in order being mixed with the changes in content. Usually we do one patch for changin" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/911839 (https://phabricator.wikimedia.org/T335388) (owner: 10Wfan) [07:19:27] 10Fundraising-Backlog: Formatting issue in 'Renew Your Recurring' email sent from Civi - https://phabricator.wikimedia.org/T337919 (10MSuijkerbuijk_WMF) Great, thanks @Cstone [07:26:54] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/926353 (owner: 10L10n-bot) [07:44:08] 10Fundraising-Backlog: Formatting issue in 'Renew Your Recurring' email sent from Civi - https://phabricator.wikimedia.org/T337919 (10krobinson) Thanks @Cstone! I'll leave this open until we see a few replies from donors for recently deployed emails (may take a few days), just in case anything else is noted, b... [10:09:03] (03CR) 10Raimond Spekking: [C: 03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/926353 (owner: 10L10n-bot) [13:30:34] (03PS41) 10Damilare Adedoyin: IP Velocity Filter: Increase count after first run [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/859168 (https://phabricator.wikimedia.org/T323544) (owner: 10AndyRussG) [13:32:44] (03CR) 10CI reject: [V: 04-1] IP Velocity Filter: Increase count after first run [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/859168 (https://phabricator.wikimedia.org/T323544) (owner: 10AndyRussG) [14:18:29] (03PS42) 10Damilare Adedoyin: IP Velocity Filter: Increase count after first run [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/859168 (https://phabricator.wikimedia.org/T323544) (owner: 10AndyRussG) [14:20:10] (03CR) 10CI reject: [V: 04-1] IP Velocity Filter: Increase count after first run [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/859168 (https://phabricator.wikimedia.org/T323544) (owner: 10AndyRussG) [14:21:06] (03PS43) 10Damilare Adedoyin: IP Velocity Filter: Increase count after first run [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/859168 (https://phabricator.wikimedia.org/T323544) (owner: 10AndyRussG) [14:31:50] 10Fundraising-Backlog: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10MBeat33) [14:32:35] 10Fundraising-Backlog: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10MBeat33) [14:33:25] 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10MBeat33) [14:59:07] 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10Pcoombe) Hi @MBeat33, are there any similar cases for Mexico? Or is it just Brazil? [15:07:26] 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10EMartin) @Mbearden @Pcoombe I saw this in testing for BR and MX as I shared in Asana. [15:21:32] 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10MBeat33) @Pcoombe there are 967 in MXN via Adyen this week [15:32:14] 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10Pcoombe) utm_source on cid=1911896 indicates it was through Google Pay, which is still supposed to be using Adyen right? [15:33:34] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Email: Civi to Acoustic Export: Job Status Failed - https://phabricator.wikimedia.org/T338041 (10LMedley-WMF) [15:34:50] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Email: Civi to Acoustic Export: Job Status Failed - https://phabricator.wikimedia.org/T338041 (10LMedley-WMF) [15:41:25] 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10Pcoombe) This query should be finding all trxn_id's which start with "ADYEN" and grouping by method. Only Apple Pay and Google Pay, and they seem to match the numbers given above. `la... [15:42:57] 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10Pcoombe) If these methods are resulting in unexpected fees, perhaps we should consider disabling them for these countries? [15:48:43] @frtech. Hello FRTech, there is an issue in flight where we are seeing Brazil and MX transactions clearing through Adyen when they should be clearing through Dlocal. https://phabricator.wikimedia.org/T338038. This has an unbreak now feel to it as donors are being charged fees for the foreign transactions clearing through the US acquirer. Can someone have a look at this quickly? We are thinking of a) taking down [15:48:43] the campaign and b) taking down the payment methods in question. Please chime in to this task. [15:50:32] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10greg) p:05Triage→03Unbreak! Setting UBN! per Evelyn in IRC, moving to Chaos Crew board. Standup is in 10 minutes to discuss. [15:50:48] thanks emartin_, we'll discuss in standup in 10 minutes [15:58:39] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10EMartin) Here are the transactions over the last 7 days that have cleared for Mexico through Adyen (excluding today): {F37091277} Here are the trans... [16:01:18] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10Cstone) On two I looked at in the console it shows Visa and MC as the Payment method but when you click into them it says Card type Apple Pay - Visa,... [16:08:35] 10Fundraising-Backlog, 10fundraising-tech-ops: migrate Silverpop export job from civi1001/frdev1001 to civi1002+frdb1006 - https://phabricator.wikimedia.org/T337498 (10Jgreen) 05Open→03Resolved a:05Jgreen→03Ejegg Yay! Thanks for all your work on this @Ejegg. I checked the process-control logs and it lo... [16:08:37] 10Fundraising-Backlog, 10fundraising-tech-ops: Migrate process-control jobs from civi1001 to civi1002 - https://phabricator.wikimedia.org/T332632 (10Jgreen) [16:11:09] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10EMartin) FRtech, I wanted to also point out that we are seeing decent VI/MC traffic through Dlocal as well over the last week as follows:{F37091287} [16:18:08] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10Cstone) @EMartin in that report you pulled ^ does it let you pull the card type along with the payment method? I'll look from our site too but I wond... [16:20:00] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10greg) So just to be explicit, it appears this is working as intended; apple pay/gpay don't go through dlocal and instead use adyen. [16:27:28] 10Fundraising-Backlog: Migrate phab task description to cowsay format - https://phabricator.wikimedia.org/T337046 (10Cstone) {meme, src=seal-of-approval} [16:28:08] 10Fundraising-Backlog: Migrate phab task description to cowsay format - https://phabricator.wikimedia.org/T337046 (10AKanji-WMF) 05Open→03Resolved [16:29:48] 10Fundraising-Backlog: Fix India form text cosmetics before pretest - https://phabricator.wikimedia.org/T337520 (10AKanji-WMF) Adding to next sprint, will be able to triage more Tuesday. [16:29:51] 10fundraising-tech-ops: encrypt fundraising database client->server communication - https://phabricator.wikimedia.org/T170321 (10Jgreen) [16:36:24] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10EMartin) Why do we see VI and MC in Adyen through these markets? Are you saying they are AP and GP transactions and not direct credit card we re see... [16:53:22] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10EMartin) @greg @Cstone @MBeat33 @Pcoombe @spatton @RKumar_WMF All, it seems we have no issue here, other than Adyen is flagging GPay/AP transactions... [16:53:26] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10EMartin) Thanks for the quick attention here all. [17:01:56] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10greg) Glad it wasn't anything more serious. Thanks to all for the attention. [17:07:42] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10MBeat33) Thanks @EMartin and all for the clarifications, and +1 @greg [17:32:52] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10Pcoombe) @EMartin I have removed those methods from banners ([diff](https://meta.wikimedia.org/w/index.php?title=MediaWiki:FundraisingBanners/Localiz... [17:33:07] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10Pcoombe) 05Open→03Resolved a:03Pcoombe [17:34:20] 10Fundraising-Backlog: Fix India form text cosmetics before pretest - https://phabricator.wikimedia.org/T337520 (10EMartin) @spatton fyi [17:36:02] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Email: Civi to Acoustic Export: Job Status Failed - https://phabricator.wikimedia.org/T338041 (10Dwisehaupt) @LMedley-WMF We have looked into this and it appears that a maintenance mode flag got in the way of letting this job run last night. We hav... [17:47:18] 10Fundraising-Backlog, 10fr-donorservices: Double check where we should use Apple Pay and Google Pay - https://phabricator.wikimedia.org/T338058 (10Pcoombe) [17:59:13] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10EMartin) @Pcoombe Thanks so much for the fast effort here on a Friday night! Many, many thanks! [18:03:07] 10Fundraising-Backlog, 10fr-donorservices: Double check where we should use Apple Pay and Google Pay - https://phabricator.wikimedia.org/T338058 (10EMartin) @Pcoombe I am going to get an assessment from leadership on what fees we want to allow our donors to tolerate because it will all be cross border outside... [18:10:08] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Email: Civi to Acoustic Export: Job Status Failed - https://phabricator.wikimedia.org/T338041 (10KHaggard) Thanks @Dwisehaupt ! We have emails scheduled to go out on Monday so I'm hoping we can resolve this today. We didn't receive any files today... [18:36:57] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Email: Civi to Acoustic Export: Job Status Failed - https://phabricator.wikimedia.org/T338041 (10Dwisehaupt) @KHaggard We have @cstone and @jgleeson working on that right now. They will be able to fill you in shortly. [20:24:38] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Email: Civi to Acoustic Export: Job Status Failed - https://phabricator.wikimedia.org/T338041 (10KHaggard) 05Open→03Resolved a:03KHaggard Thanks to the work of @Cstone and @jgleeson and @AnnWF we're back in business! Copying the job data summ... [20:56:32] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Brazil card donors routing to Adyen? - https://phabricator.wikimedia.org/T338038 (10spatton) +1, thank you @Pcoombe for holding down the banner fort as only you can. I appreciate everyone's work on this today! [22:21:59] 10fundraising-tech-ops: Deactivate fundraising accounts for andyrussg - https://phabricator.wikimedia.org/T338075 (10Dwisehaupt) [22:23:34] 10fundraising-tech-ops: Deactivate fundraising accounts for andyrussg - https://phabricator.wikimedia.org/T338075 (10Dwisehaupt) [22:23:50] noooooooooooooooo [22:24:12] ahh I mean it's ok, so it goes [22:24:18] giant hugs to all!!!!!!!!!!!!!!!!!!!!!!!!!! [22:30:24] yeah. all just standard tracking things, but still :( [22:45:47] AndyRussG|ish: I hope you have a fun journey ahead [22:46:21] thanks so so much bd808! it's been a pleasure :) :) [22:46:59] dwisehaupt: thanks so much once again :) [22:47:10] Please come back as a volunteer and rewrite all the ugly bits of CentralNotice to be better! ;) [22:47:28] oki I'm in! [22:48:16] just-volunteer hat landing on head in a matter of hours..... [23:01:27] (03CR) 10Wfan: [C: 03+2] "Works fine locally" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/864883 (https://phabricator.wikimedia.org/T323657) (owner: 10Ejegg) [23:01:40] (03PS3) 10Wfan: Add option to donation QC to wait for messages [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/864883 (https://phabricator.wikimedia.org/T323657) (owner: 10Ejegg) [23:32:57] (03PS7) 10Wfan: Clean payment submethods and add missing payment images [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/911839 (https://phabricator.wikimedia.org/T335388) [23:35:59] (03CR) 10Wfan: Clean payment submethods and add missing payment images (031 comment) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/911839 (https://phabricator.wikimedia.org/T335388) (owner: 10Wfan) [23:50:20] (03PS1) 10Wfan: Make the submethods alphabetical [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/926547 (https://phabricator.wikimedia.org/T335388) [23:52:10] (03CR) 10CI reject: [V: 04-1] Make the submethods alphabetical [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/926547 (https://phabricator.wikimedia.org/T335388) (owner: 10Wfan) [23:52:59] (03PS2) 10Wfan: Make the submethods alphabetical [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/926547 (https://phabricator.wikimedia.org/T335388) [23:54:36] (03CR) 10CI reject: [V: 04-1] Make the submethods alphabetical [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/926547 (https://phabricator.wikimedia.org/T335388) (owner: 10Wfan)