[01:41:20] eileen: does this batch token update script look OK? https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/724785 [01:42:47] ejegg: looking [01:48:04] ejegg: thoughts on the civi update? [01:49:41] shoot, just goty handed an unhappy baby [01:49:44] back soon [02:07:08] bah sorry I was going to look at that too ejegg [02:21:01] I just looked & stepped through the batch part & it all seems fine - I got an error that was config related [02:21:22] SmashPig\Core\ConfigurationKeyException: Configuration key 'accounts/WikimediaDonations/ws-api-key' [02:22:02] im testing with an old adyen recurring id and it did switch out the token and fill the invoice ID but im failing at charging it again [02:22:22] cstone: so my take on it is that the bit that is new - the batch wrapper - seems to work ok [02:22:42] ok cool [02:22:47] (03CR) 10Eileen: Batch update payment tokens (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724785 (owner: 10Ejegg) [02:26:48] (03CR) 10Eileen: [C: 03+1] "I walked through the batch wrapper & it worked - I couldn't step through the update - but that part is unchanged. The variable name I ques" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724785 (owner: 10Ejegg) [02:27:16] cstone: well I gave it +1 - do you want to double check & raise me 1? [02:27:41] yeah im trying to run it with a bunch of recurrings in my db and getting errors but could be my bad data [02:30:52] cleaning everything out and trying again [02:34:56] cstone: I guess you are like me though - in that the changed part works - its the old part that is the problem :-) [02:35:22] i got the old part to work before which makes me think its my data [02:35:30] it just behaved on a new apple pay let me add a couple different types [02:36:40] also FYI cstone & ejegg if you use drush @wmff that does the -u part for you on dev & prod now [02:38:23] cstone: also - we are now watching 'sword art online' - Jack is so serious about it that when I *only* watched one on Tues because I was busy he wanted to watch 3 last night instead of our normal 2 as a catch up [02:38:36] haha [02:38:56] i didnt make it past like episode 5 of that show :P [02:43:12] I'm assuming I must be on episode 9000 by now [02:43:24] theres at least like 5-6? seasons or more [02:43:30] yeah! [02:43:43] i dont know why they keep playing games :P [02:46:25] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724513 (owner: 10Eileen) [02:46:58] ok cool the script behaved as expected with old adyen, coverting to new, then charging on new token [02:47:11] yay [02:47:37] cstone: if you have brain-space left 724513 ^^ should be an easy cleanup that will save brain power later [02:48:13] is there a way to test without destroying my database? :P [02:50:38] (03CR) 10Ejegg: "Oh right, I'm probably misusing that rowCount, huh?" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724785 (owner: 10Ejegg) [02:50:48] (03CR) 10Cstone: [C: 03+2] "Looks good! Tested with an apple pay, cc, and old adyen cc. It ran and converted the old adyen cc with this output:" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724785 (owner: 10Ejegg) [02:51:00] oh haha [02:51:43] thanks for the cr cstone ! [02:51:50] no worries about that rowCount [02:51:56] it counted the 1 fine at least [02:51:58] it's probably fine to set it directly [02:52:25] anyway, with it set like this we know that when it returns zero there are no more old tokens to update [02:59:57] (03CR) 10Cstone: [C: 03+2] Fix our message template install to do both reserved & editable [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724513 (owner: 10Eileen) [03:00:27] ejegg: I dunno if it's a misuse - at most I would have suggested adding a comment [03:01:31] (03Merged) 10jenkins-bot: Batch update payment tokens [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724785 (owner: 10Ejegg) [03:01:59] ejegg: i was looking at the new adyen ideal flow and unless im missing something i dont think it ever goes to the pending table [03:03:50] oh dang, we really should send iDEAL details to pending since they're leaving our site [03:04:09] OK, we might need to add a line in the new adyen adapter in DI to send that [03:04:31] it just gets the complete status so goes to donations queue [03:10:59] (03Merged) 10jenkins-bot: Fix our message template install to do both reserved & editable [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724513 (owner: 10Eileen) [03:11:57] (03PS1) 10Ejegg: Make payments listen on additional http port [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/724864 (https://phabricator.wikimedia.org/T291577) [03:18:52] cstone: shoot, it looks like we should hit this line - https://phabricator.wikimedia.org/diffusion/EDOI/browse/master/gateway_common/donation.api.php$35 [03:19:32] ah okay let me check what my local does there [03:19:43] does $authorizeResult->requiresRedirect() return true up in the adyen checkout adapter? [03:19:53] sorry, i should try that locally [03:22:16] ejegg: on the civi upgrade - do you think we should do it tomorrow (am assuming no one thinks tonight) [03:23:29] eileen: well, we should do it in this outage window! [03:23:35] tomorrow would work best for me [03:23:45] i guess i was going on that my pending table was empty but maybe its being cleared out when i didnt realize it [03:23:48] oh i know why [03:23:53] consumer? [03:24:08] yepyep sorry for the confusion [03:24:18] ejegg: cool - it's up for review anyway - but I haven't enabled the new stuff - still working through that a little [03:24:20] cool, i just tried locally and it's definitely putting a message in the queue [03:24:28] yeah mine has 63 haha [03:24:56] yep yep, we have a task for QC cron on the dev setup, right? [03:25:19] yeah and i wanted it turned off then forgot to run the pending hah [06:24:42] (03PS1) 10Eileen: Add recurring example [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/724868 [06:59:41] (03CR) 10jerkins-bot: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/724897 (owner: 10L10n-bot) [07:03:50] (03CR) 10Raimond Spekking: [C: 03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/724897 (owner: 10L10n-bot) [15:51:32] oops fr-tech i might miss standup [15:54:00] ejegg: oki no worries eh :) [16:55:29] 10fundraising-tech-ops: Fundraising access request for Sarah Guseman (Endowment Assistant - Contractor) - https://phabricator.wikimedia.org/T292212 (10Zabe) [16:56:21] 10fundraising-tech-ops: Fundraising access request for Sarah Guseman (Endowment Assistant - Contractor) - https://phabricator.wikimedia.org/T292212 (10JAdams) [17:16:21] 10Fundraising-Backlog: Ingenico payments service provider SSL certificate renewal notice - https://phabricator.wikimedia.org/T292216 (10EMartin) [17:46:43] 10fundraising-tech-ops: Fundraising access request for Sarah Guseman (Endowment Assistant - Contractor) - https://phabricator.wikimedia.org/T292212 (10Dwisehaupt) [17:47:20] damilare: AndyRussG fancy catchin up to talk through translation stuff? [17:47:21] 10fundraising-tech-ops: Fundraising access request for Sarah Guseman (Endowment Assistant - Contractor) - https://phabricator.wikimedia.org/T292212 (10Dwisehaupt) a:03Dwisehaupt [17:47:51] jgleeson, AndyRussG sorry just realised I didn't share the link earlier [17:48:10] jgleeson damilare yes one sec :) [17:59:54] 10fundraising-tech-ops: Fundraising access request for Sarah Guseman (Endowment Assistant - Contractor) - https://phabricator.wikimedia.org/T292212 (10Dwisehaupt) SSL client certificate created and sent via email. Password sent via SMS. CiviCRM account created and granted permissions. Account creation email sent. [18:00:25] 10fundraising-tech-ops: Fundraising access request for Sarah Guseman (Endowment Assistant - Contractor) - https://phabricator.wikimedia.org/T292212 (10Dwisehaupt) [18:03:12] 10Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for Janna Layton - https://phabricator.wikimedia.org/T292222 (10EYener) [18:15:11] cstone: can I update the token on one of your old recurrings and test a charge? [18:22:09] (03PS2) 10Ejegg: Update custom field table to reflect field updates. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724582 (https://phabricator.wikimedia.org/T288721) (owner: 10Eileen) [18:22:15] (03CR) 10Ejegg: [C: 03+2] Update custom field table to reflect field updates. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724582 (https://phabricator.wikimedia.org/T288721) (owner: 10Eileen) [18:29:34] Sure ejegg [18:29:40] There should be 2 old adyens [18:31:54] thanks! [18:37:05] (03CR) 10Ejegg: "Looks good, a couple of questions inline" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724358 (owner: 10Eileen) [18:42:42] (03Merged) 10jenkins-bot: Update custom field table to reflect field updates. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724582 (https://phabricator.wikimedia.org/T288721) (owner: 10Eileen) [18:55:35] cstone: cstone ok, looks like contribution_recur_id 587147 succeeded! [18:55:41] nice! [18:56:14] i'm going to run a small batch of token updates, like 5 , just to make sure the batchiness works [18:57:10] oh wait, the batch script isn't deployed yet [18:57:14] will deploy that first [18:57:23] hmm, do we deploy the 5.43 update today? [18:57:28] haha guess i had 3 adyen recurrings [18:58:00] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/725091 [18:58:13] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/725091 (owner: 10Ejegg) [18:59:14] lol there are 27 contribution_recur rows associated with either your battab or your wm.o address [18:59:18] i hope a lot of those are cancelled [18:59:35] haha a good number are [18:59:39] although that 3.33 is a mystery [18:59:45] ejegg: dwisehaupt should we push out the civi update [18:59:45] where the last charge on it was feb 2021 [19:00:00] eileen: it's looking good locally so far [19:00:03] but maybe i did something weird to it [19:00:18] ejegg: cool - I'm just looking at updated to latest patches merged [19:00:28] eileen: i'm fine with that. [19:00:54] dwisehaupt: what are people's expectations at the moment in terms of outages / notice [19:02:00] i haven't given the all clear so they should still expect outages. [19:02:32] cool [19:05:04] (03PS4) 10Eileen: 5.43.alpha [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/723806 [19:05:15] ejegg: slight update ^^ - just since-merged changes [19:08:14] 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Tech-Analytics: Create a table showing failed attempted recurring charges per month - https://phabricator.wikimedia.org/T291214 (10EYener) We should collaborate after code freeze in Q3 with @Cstone to turn logfiles into more usable raw files. Can we follow... [19:29:47] 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Tech-Analytics: Create a table showing failed attempted recurring charges per month - https://phabricator.wikimedia.org/T291214 (10Ejegg) I think it would also be possible to create rows in the contributions table with a 'failed' status. [19:35:04] 10Fundraising-Backlog, 10Analytics-Radar, 10Product-Analytics, 10Wikipedia-iOS-App-Backlog, and 2 others: Understand impact of Apple's Relay Service - https://phabricator.wikimedia.org/T289795 (10Pelagic) > @Base: “but blocks under No open proxies policy indeed are generally for both logged in and anonymou... [19:40:51] 10Fundraising-Backlog, 10Analytics-Radar, 10Product-Analytics, 10Wikipedia-iOS-App-Backlog, and 2 others: Understand impact of Apple's Relay Service - https://phabricator.wikimedia.org/T289795 (10AntiCompositeNumber) The specific reasons for hardblocking are a bit [[https://en.wikipedia.org/wiki/Wikipedia:... [19:42:23] (03CR) 10Eileen: "@ejegg I'm getting confused - this is only run from CI isn't it? We can think about it as our window into configuring what CI does?" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724358 (owner: 10Eileen) [19:42:29] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724358 (owner: 10Eileen) [19:44:45] (03PS2) 10Eileen: Clone buildkit [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724358 [19:44:50] (03CR) 10Ejegg: Clone buildkit (032 comments) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724358 (owner: 10Eileen) [19:45:14] oops, sorry, i replied too fast [19:45:20] i see you got rid of that line [19:46:52] ejegg: ok - I was wondering if the workspace stuff is out of date too [19:47:41] potental removal for lines around 9 [20:02:56] 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Performance-Team (Radar): Reduce module registration overhead from CentralNotice - https://phabricator.wikimedia.org/T221805 (10DannyS712) Looked into this separately as part of general reduction of modules in a bunch of extensions. The following... [20:05:11] (03CR) 10DannyS712: "This change is ready for review." [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/724983 (https://phabricator.wikimedia.org/T221805) (owner: 10DannyS712) [20:08:16] (03CR) 10jerkins-bot: [V: 04-1] Merge single use dependencies into ext.centralNotice.adminUi [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/724983 (https://phabricator.wikimedia.org/T221805) (owner: 10DannyS712) [20:09:20] 10Fundraising-Backlog, 10FR-Tech-Analytics: Collaboration on RML data flow - https://phabricator.wikimedia.org/T292247 (10EYener) [20:13:03] ejegg: are you still checking the civi update? [20:13:27] eileen: trying to do a couple token updates actually [20:13:34] sorry, lemme pull down that last PS [20:13:55] let's see if any of the changes need me to redo the DB updates... [20:14:20] I think one might change the civicrm_contribution_recur.modified_date to timestamp [20:14:29] ahh, contribution_recur_modified_date type changed? [20:14:35] (03PS2) 10Eileen: Add recurring example [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/724868 [20:14:52] i'll just do that alter then [20:15:40] yeah - I'm kinda inclined to think you should blind merge that follow on too - it only affects our recurring fail mail but I think it will reduce the moving parts for the next round of review ( [20:15:51] ok [20:16:35] (03CR) 10DannyS712: "Failure is unrelated, T292251" [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/724983 (https://phabricator.wikimedia.org/T221805) (owner: 10DannyS712) [20:17:36] hmm, i don't have 'Cash' in my 'Financial Type' dropdown any more [20:17:44] eileen: is that part of this update? ^^^ [20:18:21] Isn't that what we generally use for donations? [20:19:09] ok, it was there but just disabled. huh [20:19:23] I think it's unrelated [20:20:21] oh hmm, Direct Mail Appeal dropdown is pretty reduced too [20:24:42] ejegg: I'll do a rebuild on mine & check - those drop downs look fine atm locally [20:25:54] (03CR) 10Ejegg: [C: 03+2] "Looks pretty good! I don't see any big breakage locally - just some financial type and Direct Mail Appeal options disabled. Let's click ar" [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/723806 (owner: 10Eileen) [20:30:19] ejegg: so the buildkit one passed - the theory is we could merge that & then they could remove their git clone - because the conditionals in ours would make it work either way? [20:32:40] oh yeah, sounds good eileen [20:33:09] (03PS3) 10Ejegg: Clone buildkit [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724358 (owner: 10Eileen) [20:33:15] ejegg: did you also see that I'm thinking that maybe instead of getting rid of the civicrm symlink we actually ditch the whole repo [20:33:22] (like we did for drupal) [20:33:26] (03CR) 10Ejegg: [C: 03+2] "Nice!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724358 (owner: 10Eileen) [20:33:26] and check the code in [20:33:34] i see that eileen [20:34:13] does seem a bit fiddly for passing patch files back and forth from a full core checkout and our submodule [20:34:17] which i do sometime [20:34:18] s [20:39:30] ejegg: yeah - I do that too - e.g with extensions- I still have a .git directory in my extension folders & use it to engage with upstream / other repos. I just then cd up a level to commit to us [20:40:09] (03Merged) 10jenkins-bot: 5.43.alpha [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/723806 (owner: 10Eileen) [20:40:26] for civicrm we could instantiate that .git in our build - maybe the extensions too [20:40:26] eileen: ahh, i guess that would be easier than a separate checkout and passing patch files [20:40:36] yeah [20:40:48] I mean I have a separate checkout - it's dmaster [20:41:05] so I normally use that when putting stuff upwardds [20:43:01] but for pulls I have use the wmf site [20:43:31] (03PS1) 10Eileen: Submodule update - 5.43.alpha [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725109 [20:43:43] (03CR) 10Eileen: [C: 03+2] Submodule update - 5.43.alpha [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725109 (owner: 10Eileen) [20:46:12] (03Merged) 10jenkins-bot: Clone buildkit [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/724358 (owner: 10Eileen) [20:50:14] 10Fundraising Sprint Esperantoland, 10Fundraising Sprint File Systems Stage Show, 10Fundraising Sprint Git Rebase Jump, 10Fundraising Sprint Humongous bacteria petting zoo, and 15 others: Update Fundraising tech CI image to use upstream buildkit, no symlink for civ... - https://phabricator.wikimedia.org/T277500 [20:50:42] ejegg: I commented back to hashar on the phab - wondering if I should ask him to stop checking out the civi repo at the same time [20:50:50] 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Tech-Analytics: Create a table showing failed attempted recurring charges per month - https://phabricator.wikimedia.org/T291214 (10Ejegg) See T222220 [20:51:18] dwisehaupt: I'm about 10 mins away from having the civi update merged through to deployment (waiting on ci) [20:52:28] coolio. need anything from me? [20:54:54] (03Merged) 10jenkins-bot: Submodule update - 5.43.alpha [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725109 (owner: 10Eileen) [20:56:08] dwisehaupt: so I guess we should turn off queues & maybe notifiy another brief outage [20:56:25] (we could get by without an outage but since it's the maintenance window) [20:57:10] ok. i'll turn off the queues now. [20:58:19] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment + 9aba2fa7331ec7539d74bb4a18cbbc069c5dfeec Fix our message template install to do both reserved & editable + e1523a82d2ff20533079bd1cf12ee0a6563d7a62 Batch update payment tokens + f0fca1de29defe02b19cc4c16179ccff83b0fd24 Update custom field table to reflect field updates. + 52df1ae0f19a90e9f4eb68b7526c9a9be2 [20:58:19] update - 5.43.alpha [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/725113 [20:58:48] 10fundraising-tech-ops: Configure cross host data sync for privatebin data - https://phabricator.wikimedia.org/T283092 (10Jgreen) 05Open→03Resolved p:05Triage→03Medium a:03Jgreen This is done, on a 6H interval. [20:58:52] 10fundraising-tech-ops, 10Patch-For-Review: Reconfigure fundraising public-reporting server role to fundraising-data - https://phabricator.wikimedia.org/T255435 (10Jgreen) [20:59:18] ok. p-c jobs stopped on civi and frdev. [20:59:25] (03PS2) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/725113 [20:59:51] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/725113 (owner: 10Eileen) [21:00:11] folks are aware of the main window so we don't need to notify. i can put civi in maint mode if we want to ensure no one makes changes. [21:00:21] ok let's do that then [21:00:33] ejegg: the deploy includes your update patch [21:00:46] yep, sounds good [21:00:52] i just tested the individual updates [21:01:03] and once this is out I'll try some batches [21:02:25] ok. maint mode switch flipped for civi1001. [21:02:53] ejegg: how do you feel about blind merging this - https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/civicrm/+/724868 - I think it's gonna be easier to handle the moving parts on the recurring fail message if we get that in - but it's not really testable in itself right now [21:03:07] oh sure, sorry, I meant to do that earlier [21:03:09] (03CR) 10Ejegg: [C: 03+2] Add recurring example [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/724868 (owner: 10Eileen) [21:03:17] thanks [21:06:30] !log civicrm revision changed from 2ecb8f0bcd to d4da344274, config revision is 77cb7ec866 [21:06:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:13:34] ok upgrade is slower than expected because of those changes to contribution recur - but won't be that slow [21:13:57] (03Merged) 10jenkins-bot: Add recurring example [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/724868 (owner: 10Eileen) [21:14:57] (03PS1) 10Eileen: Submodule update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725115 [21:15:08] (03CR) 10Eileen: [C: 03+2] Submodule update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725115 (owner: 10Eileen) [21:25:52] (03Merged) 10jenkins-bot: Submodule update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725115 (owner: 10Eileen) [21:35:16] query still running [21:35:36] cool. [21:37:14] it just finished [21:38:32] heh. yeah. see that on the replication lag graphs. [21:41:53] cool [21:42:03] I guess we can start queues again now [21:42:50] ok. sounds good to me. [21:42:58] reenable civi access too? [21:47:09] ok. we are back in business. [21:48:29] cool cool. I started batch updating some tokens in the background but hit one which gets an error in the payment processor API lookup [21:48:46] and I didn't account for how to skip problematic ones in the script [21:49:26] i think the error 'no associated saved details' means it won't actually be possible to change, so maybe I just cancel the recurring [21:49:39] gonna try to find them in the console [21:52:53] ejegg: let me know if you want help looking [21:56:15] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1225 [21:57:09] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1283 [21:57:15] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1288 [21:57:41] ^^ those are expected. i'm acking them on the paging side but leaving icinga since they should clear soon. [22:00:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1466 [22:01:15] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1525 [22:02:09] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1583 [22:02:15] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1588 [22:04:25] thanks cstone [22:05:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1766 [22:06:15] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 3018509 Threads: 12 Questions: 167720795 Slow queries: 1458 Opens: 3727 Flush tables: 1 Open tables: 1087 Queries per second avg: 55.564 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [22:07:09] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 2952206 Threads: 11 Questions: 166855939 Slow queries: 1400 Opens: 3934 Flush tables: 1 Open tables: 1165 Queries per second avg: 56.519 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [22:07:15] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1888 [22:10:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2066 [22:12:09] cstone: oh shoot, I forgot to filter on status [22:12:15] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 2678954 Threads: 11 Questions: 158762890 Slow queries: 1362 Opens: 648193095 Flush tables: 1 Open tables: 200 Queries per second avg: 59.263 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [22:12:24] cstone: that one was cancelled on max failures [22:12:32] i'll have a lil patch up in a few [22:12:39] ah okay [22:13:28] dbs looking good. [22:15:15] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 289092 Threads: 15 Questions: 4163435 Slow queries: 236968 Opens: 28446457 Flush tables: 1 Open tables: 200 Queries per second avg: 14.401 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [22:19:10] (03PS1) 10Ejegg: Limit batch token update to In Progress [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725149 [22:19:45] oops, telling me that's an invalid field [22:23:28] (03PS2) 10Ejegg: Limit batch token update to In Progress [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725149 [22:23:45] cstone: do you have a sec to look at ^^^? [22:23:54] yep yep [22:23:55] I know I owe you some CR on your SmashPig patch too... [22:24:38] that ones still not 100% too the tests are all broken hah [22:25:52] gerrit misbehaving for me boo [22:28:13] oh right, existing tests need updates [22:29:36] the tests for the smashpig patch i have i meant [22:31:03] hmm I still get the invalid field or was I looking at the not latest [22:31:37] ejegg: maybe your 2nd patch didnt get the change? [22:31:43] oh derp, lemme look [22:31:56] hah, it didn't [22:32:47] (03PS3) 10Ejegg: Limit batch token update to In Progress [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725149 [22:32:59] ok cstone ^^^ should be for real [22:34:13] hmm, i guess the process could actually cancel donations that it can't find saved details for [22:35:04] hmm would we accidentally cancel ones that worked on old but not on new for some reason? [22:36:06] ah i was thinking it would be ones that are in a broken state at the console but which we hadn't tried to charge yet [22:37:14] I guess we can try with the limit patch and see if we run into more trouble [22:38:37] is that invoice_id IS NULL the right way to do that its getting caught on mine when i have something in there, but I also have bad data too [22:42:24] cstone the old ones all had NULL invoice_id [22:42:31] while the new ones have something in there [22:42:56] that's the recur.invoice_id, not the regular contribution one [22:43:20] yeah I think its breaking earlier in the chain I was trying to fudge the data [22:47:29] ok I don't know what happened locally but that patch update is good [22:47:38] (03CR) 10Cstone: [C: 03+2] Limit batch token update to In Progress [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725149 (owner: 10Ejegg) [22:48:41] thanks! [22:50:40] (03CR) 10Ejegg: "Sorry to get to this so late! A little confused as to why it was needed - see inline comment." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/719681 (https://phabricator.wikimedia.org/T290106) (owner: 10Cstone) [22:51:35] ejegg: for ^ i didn't see the payment_method anywhere why i added it [22:53:06] cstone: so why does the ApplePayPaymentProvider need payment_method in its $params to know that it's apple pay? [22:53:38] i just don't see why the smashpig level classes would need that param if each class only deals with one payment_method [22:53:39] it was for the recurring refactor part maybe should have been a separate patch but I was trying to not replicate [22:54:02] all three use similiar recurring things but applepay+cc needed more than ideal [22:56:31] hmm, but all three of those payment providers already have their own createPayment call [22:57:46] i guess how else to get the payment method in the smashpig api level [22:57:58] and e.g. the CardPaymentProvider knows that it should send $api->createPaymentFromToken a payment_method of 'scheme', without the DI or CRM code passing in a payment_method to the CardPaymentProvider::createPayment call [22:59:02] are there any *PaymentProvider classes that work for multiple payment_method values? [23:01:18] sorry, lemme look in gerrit to see if the refactor where you needed that param is up as WIP [23:01:53] Hey sorry lost internet on phone now [23:02:45] oh i see, it's https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/SmashPig/+/717712/3/PaymentProviders/Adyen/Api.php#501 [23:02:46] ok apartment doing weird things microwave beeped too but internet is back [23:03:01] ok, lemme put up a suggested patch on that [23:03:07] (03Merged) 10jenkins-bot: Limit batch token update to In Progress [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725149 (owner: 10Ejegg) [23:03:38] yeah was trying to say it was me trying to not have that code in each of the separate api calls [23:06:29] eileen: can I deploy that latest submodule update 'CiviCRM recurring failed prep' [23:06:32] ? [23:06:39] ejegg: yep [23:06:49] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/725151 [23:06:58] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/725151 (owner: 10Ejegg) [23:13:58] cstone: does iDEAL use createPaymentFromEncryptedDetails ? [23:14:11] sorry, i can look for that [23:16:37] the ideal recurring code isnt there yet [23:17:02] it uses drecitdebit [23:17:10] createDirectDebit [23:17:16] k, I think we can just add a bool param to your new fn then [23:18:36] (03PS1) 10Ejegg: Don't require payment_method for recurring auth [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/725154 [23:18:42] like so ^^^ [23:18:53] and you'd just send false for the ideal one [23:19:29] (03CR) 10jerkins-bot: [V: 04-1] Don't require payment_method for recurring auth [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/725154 (owner: 10Ejegg) [23:19:43] oops [23:21:22] (03PS2) 10Ejegg: Don't require payment_method for recurring auth [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/725154 [23:23:18] sorry to carp on this, i just really want to keep the library-level function calls (i.e. the *PaymentProvider::createPayment calls) as simple as possible, and that param feels redundant [23:23:29] cstone: argh that geocoder message is annoying me [23:23:49] ejegg: no worries [23:26:38] oh hmm, so we don't end up sending donor email or name or anything for ideal or apple. I wonder if that's gonna make things hard for Donor Relations [23:28:06] maybe we should have another helper method to add donor info to those calls [23:28:34] sending to adyen? [23:28:50] yeah, to make it easier for DR to search in their console [23:28:52] i noticed the ideal didn't have the emails but I thought it was just an ideal thing [23:28:56] for apple or ideal donors [23:29:05] it would be helpful if we could send them yeah [23:29:19] cool, I'll make a phab task for now [23:30:08] eileen: did you get any further trying to get the sql to load for the test? [23:30:41] cstone: no - I'm tempted to just hack the test out & do the upgrade & leave that for next round [23:31:45] 10Fundraising-Backlog, 10FR-Adyen, 10FR-Smashpig: Send donor info to Adyen for iDEAL and Apple Pay - https://phabricator.wikimedia.org/T292271 (10Ejegg) [23:32:44] that also works [23:34:34] !log updated Fundraising CiviCRM from d4da344274 to d74e9aa0a1 [23:34:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:39:15] ok, so that batch updater gets about 2 tokens per second updated [23:40:34] how many are you running at a time I can finish it tomorrow morning [23:41:27] i'm just doing 50 at a time, but I figure I'll make a p-c job for it [23:41:39] just looking at when the recur queue is high to avoid those hours [23:41:47] oops, gotta help with dinner [23:43:45] wouldnt they be the ones that start at utc 0 or 20 mins ? I think it usually finishes around 2-4am? [23:46:03] looks like 10-14utc over the last 7 days. [23:46:22] https://frmon.frdev.wikimedia.org/goto/wyGrD6H7k?orgId=1 [23:47:01] ignore the massive spikes you see on there since those were when the db maint was going on. [23:47:33] these wouldnt be the recurring queue though they would be the smashpig job [23:48:00] oh. do they not make it into the queue also? [23:48:28] they go into the donations queue [23:49:53] that's harder to judge as there have been emails and banner up over the last 30 days. but i don't see any bad times really. [23:50:54] the main issue would be the smashpig job and the token update are both touching the contribution_recur table [23:52:22] (03PS1) 10Eileen: Add message_admin to local installs [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725158 [23:53:20] cstone: are you ok to + 2 ^^ - I actually enabled it on live & so we should on dev too - I'm still working through some issues but it is a pain to renable on civibuild & I think it will save brain cycles [23:54:21] sure [23:55:06] (03CR) 10Cstone: [C: 03+2] Add message_admin to local installs [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725158 (owner: 10Eileen) [23:55:18] thanks! [23:55:58] so digging into the db. the only time of regular high query volume on non raising days is between 0100-0330 UTC [23:59:27] yeah that lines up where I remember the recurring charge job finishing at [23:59:30] i'm not seeing a huge reason on the db or queue side that would cause major issues.