[00:00:44] there are a mix of different slow queries over the time frame. the worst in time executing are probably the fr_stats ones [00:03:51] but there are only 2 every hour. one takes ~200 seconds and the other takes 360+ secs. [00:04:49] no real lock time on the queries, but analysing 135m rows to extract 4-5k seems like it may need a rethink. [00:17:52] is the queue still heavy - we could dial back greeting processing if we think it's relevant to test - but maybe all the other noise is too much. What I should prob do is add a patch to allow us to turn it on & off either as a batch value or by setting - [00:24:20] (03PS1) 10Wfan: WIP: Create a general function for pending table resolver to get latest payment status [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/832006 (https://phabricator.wikimedia.org/T316587) [00:24:49] (03CR) 10CI reject: [V: 04-1] WIP: Create a general function for pending table resolver to get latest payment status [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/832006 (https://phabricator.wikimedia.org/T316587) (owner: 10Wfan) [00:28:22] (03PS2) 10Wfan: WIP: Create a general function for pending table resolver to get latest payment status [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/832006 (https://phabricator.wikimedia.org/T316587) [00:30:06] yeah. queue is still at 12k. [00:30:12] sorry been cooking dinner. [00:31:20] (03PS3) 10Wfan: WIP: Add adyen to pending transaction resolver [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/831618 (https://phabricator.wikimedia.org/T316587) [00:39:05] need to run $kid to dance. back in ~1hr to check in. [00:42:48] so eileen the create contact is crazy slow but the update contact is not [00:43:16] ejegg: argh - so you were able to test that theory? [00:43:23] I mean with the greetings [00:43:25] just looking at the graph [00:43:33] ok - so we should turn off & try [00:43:42] I don't think it would be greetings in that case, but I would like to see [00:44:15] slowing all the way down to 1-2 sec for a create seems like it's gotta be the database [00:44:30] trying to think of what would make an insert that much slower than an update [00:47:11] (03CR) 10CI reject: [V: 04-1] WIP: Add adyen to pending transaction resolver [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/831618 (https://phabricator.wikimedia.org/T316587) (owner: 10Wfan) [00:48:23] ejegg: not sure - but pushing a patch to turn the greeting processing back off there - & then will follow up with one that allows us to UI it on & off [00:50:10] (03PS1) 10Eileen: Turn back off greeting processing [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/832009 [00:52:36] And I am seeing those delete deleted time out after 35 mins should I make the timeout longer? (Those longer one contains too many not able to delete contacts) [00:53:19] ejegg: ^^ [00:53:37] oh also - we turned on delete deleted contacts - that is another change that could affect performance [00:54:19] wfan: there are not that many contacts in the db that can't be deleted so why would that be? [00:56:59] we run 1000 per batch, and those send overtime failmail contains around 20 contacts that can not be deleted, but the rest batch might got 0 [01:00:06] ohhh that could totally be the reason, huh? [01:00:22] we can turn the job off for a bit [01:00:30] yes, let's do that first maybe? [01:00:34] wfan: but there are only a few hundred in the whole db & you are moving past the fails - so we shouldn't get 'that' many [01:00:36] wfan does that sound good? [01:01:02] ejegg: ok - but maybe start merging my patch too as it will take time to get through jenkins - am doing a better one as a follow up [01:01:06] eileen: ok, will do [01:01:45] (03CR) 10Ejegg: [C: 03+2] Turn back off greeting processing [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/832009 (owner: 10Eileen) [01:05:18] wfan oh hey, it would be nice if that deleted contacts job output how many contacts it touched each time [01:06:41] eileen: ohh, maybe it's not actually skipping the undeletable ones since your patch isn't merged? https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/830958 [01:07:08] oh as long as it is not time out, it's should be 1000 [01:07:40] it merged, and that's why I see some batch has 0 can not delete contact [01:11:32] let me get the time out longer to 50 to break the failmail first [01:13:18] wfan: let's turn it off for now to see if the queue speeds up [01:13:29] I think it's causing those deadlocks with the donation queue consumer [01:14:06] oh turn off? ok [01:14:13] !log disabled delete_deleted_contacts job (will take effect when current job ends) [01:14:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:14:56] wfan but it will be interesting to see how many it processes in any case, since it will be less than 1000 when there are fewer than that number in the db [01:15:08] (03Merged) 10jenkins-bot: Turn back off greeting processing [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/832009 (owner: 10Eileen) [01:15:19] (03PS2) 10Ejegg: Further fixes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/830958 (https://phabricator.wikimedia.org/T316059) (owner: 10Eileen) [01:15:47] oh i see wfan, you squashed that change into your other patch [01:15:55] I'll abandon that now-empty one then [01:16:02] ok [01:16:15] (03Abandoned) 10Ejegg: Further fixes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/830958 (https://phabricator.wikimedia.org/T316059) (owner: 10Eileen) [01:17:57] Thanks, then we can create a follow up patch to print out the touched number, should we create a different ticket for this? and once this done, turn this job back on live? [01:18:13] ok, that seems to have helped the processing rate [01:18:27] wfan i don't think we need a new ticket [01:19:01] ok, then should we move that ticket back to in progress then? [01:20:10] (03PS1) 10Ejegg: Output count of deleted contacts [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/832011 (https://phabricator.wikimedia.org/T316059) [01:20:31] wfan no need, I think it's too small a change to need phab overhead [01:20:40] ok [01:21:01] (03Abandoned) 10Eileen: Further fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/830956 (https://phabricator.wikimedia.org/T316059) (owner: 10Eileen) [01:21:04] ok, so processing rate is at least steadily up to >4 per second with that job off [01:21:43] ejegg: so how long should we give it before trying the greetings? [01:22:16] there are only 5k left in the queue, so we don't have much more time [01:22:36] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/832012 [01:22:47] probably will be done in 15-20 minutes [01:22:59] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/832012 (owner: 10Eileen) [01:23:46] ejegg: ok pushing it now then [01:23:56] cool cool [01:24:52] !log civicrm upgraded from d91b4a2c to e82d9cd0 [01:24:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:25:01] ok ejegg lessee [01:33:25] hmm not sure that moved the needle [01:34:03] well slightly... [01:41:24] hmm it's up now - [01:42:17] heh, just in time to get the queue to zero [01:42:31] so yeah - I'm making setting-tunable & we can re-test next time with a UI adjustment [01:42:37] great! [01:42:43] (03CR) 10Wfan: [C: 03+2] Output count of deleted contacts [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/832011 (https://phabricator.wikimedia.org/T316059) (owner: 10Ejegg) [01:43:04] should that deleted_contacts job have a load detector like the merge does? [01:43:11] the thing is - there is an improvement upstream that might help too - which I'll pull into the next update - so we can test with that too [01:43:18] ok, great [01:43:33] whereabouts in the pipeline are those improvements? [01:43:35] re deleted_contacts - yeah - I don't think we originally imagined it always running - not during BE [01:43:51] ah ok, so manually turning it off for tests [01:43:54] the improvemnts are merged & will be in what I push up to test this sprint [01:44:19] well we could load throttle deleted contacts - but I'm also not sure if it needs to run year-round [01:44:27] load throttling is probably good to do though [01:44:29] back. just let me know if you need anything. gotta head back out in about 30 mins for pickup but i'm around until then. [01:44:32] sorry, i meant where in the code are the improvements? [01:44:49] hi dwisehaupt , it looks like a conflicting job was the main culprit here [01:45:04] oh the improvements upstream - basically the greeting process is a little more efficient [01:45:05] but in any case, the queue is empty [01:45:11] ah cool eileen [01:47:00] AndyRussG: so we should be able to use the same mediawiki image for donatewiki as we are using for paymentswiki in fr-dev, right? [01:47:13] it would all be on the fr-dev side as far as I can tell [01:47:14] ejegg: cool. was just reading the scrollback. [01:52:30] eileen: oh hey, so when we want to do a full Silverpop export, we're still just doing a temporary deploy with different code, huh? [01:53:02] for full export - yeah - I just run stuff manually -it's a bit creepy - when does it kick off? [01:53:13] in an hour? [01:53:22] good question [01:53:37] I think what I do is recreate the export view but without the modified date filter [01:53:42] & run the last bit [01:53:56] but re the recurrings - are they updating through to the other tables? [01:53:58] oh but don't we need to run a bit more without filter? [01:54:07] yeah just thinking [01:54:09] to get updated values for has_recur etc? [01:55:00] yeah, kicks off in an hour [01:56:09] ok let's check the state of a contact that is update-due - do we had a cid? Is the modified date updating now? [01:56:33] (03Merged) 10jenkins-bot: Output count of deleted contacts [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/832011 (https://phabricator.wikimedia.org/T316059) (owner: 10Ejegg) [01:56:33] eileen: yep, it seems to be updating now [01:56:51] ok - do we have a contact who is wrong at their end but not ours? [01:56:58] let me see [02:01:19] ejegg: I wouldn't recommend using a fr-specific image for donateandbannerwiki [02:01:39] for CN I've used the stock MW image that devs working on stuff on the main cluster normally use [02:02:14] I think we can just add in the most recent version of that, and automagically stay on top of what's needed for main cluster compatibility, as well as cross-team work? [02:02:32] ok - this looks like one ejegg https://civicrm.wikimedia.org/civicrm/contact/view?reset=1&cid=1683114 [02:02:50] thanks eileen, I hadn't come across one yet [02:03:43] looks like the ones that were cancelled via that new job are correctly exported [02:04:10] ejegg: so this is what I think we need to run (I can do it ) [02:04:11] https://etherpad.wikimedia.org/p/up [02:04:47] note the two changes from the main script are highlighted - but on the first line the hightlight is just the ; [02:04:54] cos I removed the stuff before it [02:05:45] I take that back - first line now highlighted [02:06:08] after running that then I will check with `select contactid, modified_date,AF_has_active_recurring_donation FROM silverpop_export_view_full WHERE contactid=1683114` [02:06:50] & if all good we can fire up another file before or after the nightly (probably after since I think it might unsettled katie if there is no warning) [02:07:03] ok i think that seems right [02:07:15] you saw my tiny edit to the statuses? [02:09:06] AndyRussG: so for donatewiki we don't care about the http port for forwarding, I guess we can strictly serve on https. Any concern with having to manually manage xdebug settings inside the container for that one service? [02:13:39] ejegg: yep - but I think that needs to be in the original too [02:14:56] oh shoot, I thought we had done that! [02:15:20] yeah maybe [02:15:25] i might be in local git hell [02:15:39] no i don't see it locally either [02:16:08] hah oh yeah, it's there [02:16:25] and I think it's even deployed [02:16:33] wfan did a tools deploy a few days ago [02:16:42] and that change is merged to the deploy branch [02:18:35] something is crazy locally [02:19:32] ejegg: ah hmmm good point about xdebug [02:19:35] it's like a git ignore is messing with me [02:19:54] eileen ok, so short of a big rewrite I think we could at least make the @offSetInDays settable from python [02:20:23] and have python look for it on the commandline [02:20:33] to make this a little less of a hairy process [02:20:37] oh right - yeah - if it was php I would have done it a dozen times over - it's just python is a slog for me [02:20:43] ejegg: IIRC xdebug on stock images is really a pain, and for sure it'd be nice to have that consistent [02:21:04] it might have been fixed up, but it might not, and consistency yeah would still be nice [02:21:15] ok that contact did NOT update with that - will look [02:21:27] I guess we can also just re-use the payments image yeah [02:21:57] ok - silverpop export too [02:22:27] thinking about it some more that'll likely be best, also for centralization of logs, config, login prompt candy, etc [02:22:28] ah yeah [02:23:30] AndyRussG: yeah, there's nothing really payments-specific about your fundraising-mediawiki image, mostly just those niceties [02:24:19] yeah [02:24:21] oki [02:25:32] and it seems unlikely we'll really run up against something too funky wrt differences in libraries between our image and main prod [02:25:38] what about PHP version? [02:26:56] hmm 7.2.3 [02:26:59] https://en.wikipedia.org/wiki/Special:Version [02:27:32] and payments is now going to 7.4 right? [02:27:55] would there be an easy-ish way to switch donatebannerwiki to 7.2.3? [02:28:19] hmmm one other difference is the script path config [02:28:52] the whole /w/ path in the URL which IIRC is different for payments [02:30:03] compare: https://payments.wikimedia.org/index.php? etc. [02:30:23] vs https://donate.wikimedia.org/w/index.php? etc. [02:31:19] or for non-ugly URL format, https://donate.wikimedia.org/wiki/Special:LandingPage? etc. [02:31:42] not hugely important, I think it's just a bit of Apache config tweaking [02:32:05] I had also been thinking it might be nice to move the apache config out of the images and into the config dir [02:32:15] but anyway the PHP version might be more important [02:32:39] and also both can also be left for later and not included in an MVP dockerized donatebannerwiki [02:32:46] does that make sense? [02:40:25] AndyRussG: hmm, yeah, would be nice to match production more closely [02:41:17] but I guess locally I've always developed against a way different version of PHP assuming that being on something later would just be stricter, as long as I didn't try to use any unsupported features [02:41:21] hmm hmm [02:43:33] eileen: I've gotta head to bed, but maybe we can do the full export tomorrow if you don't feel like doing it solo [02:44:06] ejegg: yeah - I'm rebuilding the export table - then the nightly will kick off [02:44:31] ejegg: we could use the payments wiki image as a base and install a lower version of php alongside the stock one... maybe already included in debian? [02:44:32] so remaining is some common sense checking, communication with Katie & doing the file export part [02:44:51] ¡que descanses! [02:44:57] so the second two are pre-job anyway [02:45:15] AndyRussG: weird, MW is supposed to require PHP 7.3+ [02:45:17] https://www.mediawiki.org/wiki/Manual:Installation_requirements [02:45:50] huh that is funny! [02:46:01] and meta is on 7.4! https://meta.wikimedia.org/wiki/Special:Version [02:46:09] wow, odd [02:46:24] maybe it's a gradual rollout [02:46:36] wait, when I load meta it says 7.2.34 [02:46:46] must be gradual and I must be hitting a different backend [02:47:11] I get 7.4 on de. [02:47:18] funny [02:47:30] yeah also 7.4 on fr https://fr.wikipedia.org/wiki/Sp%C3%A9cial:Version [02:47:44] k, good night [05:32:58] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_38) - 10https://gerrit.wikimedia.org/r/832071 (owner: 10L10n-bot) [06:58:01] 10WMSE-Content-partnerships-support-2021-Fundraising, 10User-Jenny_Brandt_WMSE: Pick a CRM donation management software system for the F2F campaign 2022 - https://phabricator.wikimedia.org/T281683 (10Jenny_Brandt_WMSE) [[ https://docs.google.com/document/d/1afopEw0ON3QUVlnIMp3u7gD_cnSA0ZTvgNB4hsZUWcU/edit | No... [07:08:42] 10WMSE-Content-partnerships-support-2021-Fundraising, 10User-Jenny_Brandt_WMSE: Pick a CRM donation management software system for the F2F campaign 2022 - https://phabricator.wikimedia.org/T281683 (10Jenny_Brandt_WMSE) We are awaiting answers from CRM chapters. A reminder will be sent out. [12:24:23] 10Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for Erich Brill - https://phabricator.wikimedia.org/T317560 (10SHust) @Dwisehaupt I have added Erich to [[ https://collab.wikimedia.org/wiki/Fundraising#Contact_List | collab ]]. Thank you! [13:33:52] 10Fundraising Sprint Quarkery, 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface: Don't tokenize cards when donation is under Monthly Convert minimum - https://phabricator.wikimedia.org/T316840 (10Ejegg) [13:33:55] 10Fundraising Sprint Quarkery, 10Fundraising-Backlog: Adyen iDeal code seems to go through DirectDebitPaymentProvider - https://phabricator.wikimedia.org/T316368 (10Ejegg) [13:34:03] 10Fundraising Sprint Quarkery, 10Fundraising-Backlog, 10FR-Smashpig: When throttling failmail, send a count of suppressed emails with the next one sent. - https://phabricator.wikimedia.org/T315238 (10Ejegg) [13:34:07] 10Fundraising Sprint Quarkery, 10Fundraising-Backlog, 10FR-WMF-Audit: Clarify audit processor logging - https://phabricator.wikimedia.org/T315034 (10Ejegg) [13:34:11] 10Fundraising Sprint Quarkery, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Way to send Civi TY email to a group? - https://phabricator.wikimedia.org/T314525 (10Ejegg) [13:34:15] 10Fundraising Sprint Quarkery, 10Fundraising-Backlog: Civi emails and donations have different timestamps in the activities tab - https://phabricator.wikimedia.org/T313510 (10Ejegg) [13:34:19] 10Fundraising Sprint Quarkery, 10Fundraising-Backlog: Check need to account for opt-in param loss from 2022-06-15 through 2022-06-30 - https://phabricator.wikimedia.org/T311974 (10Ejegg) [13:34:27] 10Fundraising Sprint Quarkery, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Patch-For-Review: Added in Assistant Email for Individual Records in Civi - https://phabricator.wikimedia.org/T309503 (10Ejegg) [13:34:55] 10Fundraising Sprint Quarkery, 10Fundraising-Backlog: Exhaustively test bullseye upgrade in fundraising-dev - https://phabricator.wikimedia.org/T317701 (10Ejegg) [13:34:57] 10Fundraising Sprint Quarkery, 10Fundraising-Backlog, 10FR-Adyen, 10MediaWiki-extensions-DonationInterface: Stop Adyen adapter from sending an additional pending message on success - https://phabricator.wikimedia.org/T317713 (10Ejegg) [13:35:00] 10Fundraising Sprint Potato, 10Fundraising Sprint Quarkery, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Last CiviCRM point update for the year - https://phabricator.wikimedia.org/T316713 (10Ejegg) [13:35:02] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Quarkery, 10Fundraising-Backlog: Review steps and create subtasks for Adyen Pending Transaction Resolver - https://phabricator.wikimedia.org/T316582 (10Ejegg) [13:35:04] 10Fundraising Sprint Quarkery, 10Fundraising-Backlog, 10Epic: Upgrading payments-wiki from buster to bullseye - https://phabricator.wikimedia.org/T315585 (10Ejegg) [13:35:06] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising Sprint Quarkery, 10Fundraising-Backlog: Custom field "AF_has_active_recurring_donation" is showing false positives - https://phabricator.wikimedia.org/T315102 (10Ejegg) [13:35:08] 10Fundraising Sprint NaN is a Number, 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising Sprint Quarkery, and 2 others: Get imports running in the background - https://phabricator.wikimedia.org/T312166 (10Ejegg) [13:35:10] 10Fundraising Sprint Potato, 10Fundraising Sprint Quarkery, 10Fundraising-Backlog, 10Patch-For-Review: Use nightly rather than weekly audit file for Adyen - https://phabricator.wikimedia.org/T314753 (10Ejegg) [13:35:12] 10Fundraising Sprint NaN is a Number, 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising Sprint Quarkery, and 2 others: Email Greeting Field in Civi - blank for some donors - https://phabricator.wikimedia.org/T308181 (10Ejegg) [13:35:14] 10Fundraising Sprint NaN is a Number, 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising Sprint Quarkery, and 3 others: Creating New Matching Gift Import - https://phabricator.wikimedia.org/T303986 (10Ejegg) [13:35:16] 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising Sprint Machine Unlearning, 10Fundraising Sprint NaN is a Number, and 7 others: Implement Braintree Webhook signature validation - https://phabricator.wikimedia.org/T311169 (10Ejegg) [13:35:18] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Quarkery, 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Complete Adyen Checkout documentation to include donation statuses for all stages/permutations - https://phabricator.wikimedia.org/T316583 (10Ejegg) [13:35:20] 10Fundraising Sprint NaN is a Number, 10Fundraising Sprint Quarkery, 10Fundraising-Backlog, 10Epic: [epic] Adyen - auto settle stopped prior donors (Pending transaction Resolver) - https://phabricator.wikimedia.org/T299692 (10Ejegg) [13:35:22] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Quarkery, 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, and 3 others: Donation Form Error Message - Country Detection - https://phabricator.wikimedia.org/T316578 (10Ejegg) [13:36:22] ok fr-tech, in case you couldn't tell from that IRC flood, the new sprint is ready. Please move stuff into the right columns when you get a chance! [13:36:51] woohhooo thx ejegg|away :) [13:37:46] thanks Jeff_Green ! [13:37:51] oops ejegg|away * [14:57:38] 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising Sprint Machine Unlearning, 10Fundraising Sprint NaN is a Number, and 5 others: Create IPN listener for Braintree - https://phabricator.wikimedia.org/T303451 (10jgleeson) [14:58:43] 10Fundraising Sprint Quarkery, 10Fundraising-Backlog, 10FR-Adyen, 10MediaWiki-extensions-DonationInterface: Stop Adyen adapter from sending an additional pending message on success - https://phabricator.wikimedia.org/T317713 (10jgleeson) a:03jgleeson [15:32:52] 10Fundraising Sprint Potato, 10Fundraising Sprint Quarkery, 10FR-Adyen, 10Patch-For-Review: Write code for Adyen Pending Transaction Resolver - https://phabricator.wikimedia.org/T316587 (10AnnWF) [15:33:36] 10Fundraising Sprint Potato, 10Fundraising Sprint Quarkery, 10Fundraising-Backlog: Unit tests for Adyen Pending Transaction Resolver - https://phabricator.wikimedia.org/T316588 (10AnnWF) [15:51:14] 10Fundraising Sprint Quarkery, 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface: Don't tokenize cards when donation is under Monthly Convert minimum - https://phabricator.wikimedia.org/T316840 (10Damilare) a:03Damilare [16:42:33] (03CR) 10Abijeet Patro: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_38) - 10https://gerrit.wikimedia.org/r/832071 (owner: 10L10n-bot) [18:01:16] (03PS1) 10Cstone: Create maintenance script to batch refund Adyen payments. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/832300 [18:01:44] (03CR) 10CI reject: [V: 04-1] Create maintenance script to batch refund Adyen payments. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/832300 (owner: 10Cstone) [18:10:22] (03PS2) 10Cstone: Create maintenance script to batch refund Adyen payments. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/832300 [18:10:50] (03CR) 10CI reject: [V: 04-1] Create maintenance script to batch refund Adyen payments. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/832300 (owner: 10Cstone) [18:12:23] (03PS3) 10Cstone: Create maintenance script to batch refund Adyen payments. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/832300 [18:19:26] (03PS4) 10Cstone: Create maintenance script to batch refund Adyen payments. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/832300 [18:39:25] (03PS5) 10Cstone: Create maintenance script to batch refund Adyen payments. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/832300 [18:50:45] (03PS3) 10Wfan: Create a general function for pending table resolver to get latest payment status [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/832006 (https://phabricator.wikimedia.org/T316587) [18:57:00] (03PS4) 10Wfan: Add adyen to pending transaction resolver [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/831618 (https://phabricator.wikimedia.org/T316587) [19:11:01] (03CR) 10CI reject: [V: 04-1] Add adyen to pending transaction resolver [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/831618 (https://phabricator.wikimedia.org/T316587) (owner: 10Wfan) [20:17:54] (03CR) 10Ejegg: [C: 03+2] "Looks good, works for me!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/832300 (owner: 10Cstone) [20:18:27] (03Merged) 10jenkins-bot: Create maintenance script to batch refund Adyen payments. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/832300 (owner: 10Cstone) [20:22:09] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:2) 2 [=1], paypal_transaction_failure 31 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [20:37:13] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [21:44:00] (03CR) 10Ejegg: "just one nitpick" [extensions/FundraiserLandingPage] - 10https://gerrit.wikimedia.org/r/831243 (https://phabricator.wikimedia.org/T317427) (owner: 10AndyRussG) [22:12:58] (03CR) 10Ejegg: "Looks like this should work! Just a few minor suggestions." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/832006 (https://phabricator.wikimedia.org/T316587) (owner: 10Wfan) [22:19:11] (03PS4) 10Wfan: Create a general function for pending table resolver to get latest payment status [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/832006 (https://phabricator.wikimedia.org/T316587) [22:43:46] 10Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for Erich Brill - https://phabricator.wikimedia.org/T317560 (10Dwisehaupt) Created ssl certificate and sent via email. Created civicrm and superset accounts. Intro email with password change info has been sent. Passwords for ssl client c... [22:44:29] 10Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for Erich Brill - https://phabricator.wikimedia.org/T317560 (10Dwisehaupt)