[00:42:56] woohoo - this is passing - https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/825918 - would be good to get it merged to I can fix up patches against the old repo & we aren't dealing with change twice (I'm AFK for a couple of hours though) [01:53:44] hello.  is this the best place to ask questions about donating to wikipedia? [02:19:50] (03CR) 10Ejegg: [C: 03+2] De-symlink our CiviCRM folder [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/825918 (owner: 10Eileen) [02:31:40] (03PS3) 10Ejegg: Remove unused function [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/824312 (owner: 10Eileen) [02:31:45] (03CR) 10Ejegg: [C: 03+2] Remove unused function [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/824312 (owner: 10Eileen) [02:32:41] (03Merged) 10jenkins-bot: De-symlink our CiviCRM folder [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/825918 (owner: 10Eileen) [02:32:49] (03PS7) 10Ejegg: Remove buggy function to ensure prefix & suffix exist [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/824323 (https://phabricator.wikimedia.org/T315509) (owner: 10Eileen) [02:32:55] (03CR) 10Ejegg: [C: 03+2] Remove buggy function to ensure prefix & suffix exist [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/824323 (https://phabricator.wikimedia.org/T315509) (owner: 10Eileen) [02:34:31] yay - thanks ejegg|away - deploying now [02:40:33] (03PS1) 10Eileen: Vendor update Smash-Pig to 0.8.2.6, Donation Interface to 2.5.7.7 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/825931 [02:40:46] (03CR) 10Eileen: [C: 03+2] Vendor update Smash-Pig to 0.8.2.6, Donation Interface to 2.5.7.7 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/825931 (owner: 10Eileen) [02:42:21] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/825933 [02:42:53] (03CR) 10CI reject: [V: 04-1] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/825933 (owner: 10Eileen) [02:44:37] (03Merged) 10jenkins-bot: Remove unused function [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/824312 (owner: 10Eileen) [02:47:19] (03Abandoned) 10Eileen: Civi-stock [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/809747 (owner: 10Eileen) [02:47:25] (03Abandoned) 10Eileen: Re-add settings_location.php [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/809748 (https://phabricator.wikimedia.org/T303883) (owner: 10Eileen) [02:47:33] (03Abandoned) 10Eileen: Re-add WMF code to support our templates [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/809749 (https://phabricator.wikimedia.org/T303883) (owner: 10Eileen) [02:56:26] (03CR) 10Eileen: [C: 03+2] "recheck" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/825933 (owner: 10Eileen) [02:56:57] (03CR) 10CI reject: [V: 04-1] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/825933 (owner: 10Eileen) [02:59:54] Hmm merge to decployment failing now :-( https://integration.wikimedia.org/ci/job/wikimedia-fundraising-crm-composer-php73-docker/3754/console [03:02:47] trying cherry-pick commit [03:06:05] (03PS1) 10Eileen: Check in CiviCRM module [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/825934 (https://phabricator.wikimedia.org/T314995) [03:06:42] (03CR) 10CI reject: [V: 04-1] Check in CiviCRM module [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/825934 (https://phabricator.wikimedia.org/T314995) (owner: 10Eileen) [03:11:59] hmm ok - not sure about what is happening ^^ - any idea AndyRussG ? this is the commit that removes the submodule [03:12:14] eileen___: hiiii [03:12:17] one sec [03:12:55] note that ^^ is a huge commit - 8MB of files so it could be network - or some trace of the submodules [04:15:50] 10Fundraising Sprint Esperantoland, 10Fundraising Sprint File Systems Stage Show, 10Fundraising Sprint Git Rebase Jump, 10Fundraising Sprint Humongous bacteria petting zoo, and 19 others: Fix civicrm repo to be non-symlinked - https://phabricator.wikimedia.org/T289100 (10Eileenmcnaughton) [04:15:59] 10Fundraising Sprint NaN is a Number, 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Decommission Fundraising's crm/civicrm git repo - https://phabricator.wikimedia.org/T314995 (10Eileenmcnaughton) 05Resolved→03Open It seems I c... [04:39:14] eileen___: link to the patch for the CI change? [04:39:28] (03CR) 10AndyRussG: "recheck" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/825934 (https://phabricator.wikimedia.org/T314995) (owner: 10Eileen) [04:41:03] AndyRussG: it's linked in https://phabricator.wikimedia.org/T314995#8180151 - also note it might be out of our control [04:41:16] ok thanks! [05:05:02] still failed ^ [05:06:37] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_38) - 10https://gerrit.wikimedia.org/r/825998 (owner: 10L10n-bot) [07:26:42] (03CR) 10Abijeet Patro: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_37) - 10https://gerrit.wikimedia.org/r/825511 (owner: 10L10n-bot) [07:56:14] (03CR) 10Abijeet Patro: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_38) - 10https://gerrit.wikimedia.org/r/825998 (owner: 10L10n-bot) [12:56:53] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fr-email-preference-center: New Custom Field - Opt Out Endowment Emails - https://phabricator.wikimedia.org/T248019 (10JAdams) In Civi, we do have an Endowment Opt Out field, as well as a Planned Giving Opt out, so I think we're set here. {F35487627} [13:46:07] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Alerts: INVALID_MESSAGE Recurring donation, but no subscription ID or recurring payment token found failmail coming from the listener on ideal - https://phabricator.wikimedia.org/T315031 (10Damilare) p:05Triage→03Medium [14:05:39] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Way to send Civi TY email to a group? - https://phabricator.wikimedia.org/T314525 (10MDemosWMF) Thanks @Eileenmcnaughton ! This can be low priority. I realized we just have to mark differently in the imports to ensure that those records already in Civi wi... [14:13:08] 10Fundraising Tech - Chaos Crew: Create documentation listing FR-tech service providers and URL - https://phabricator.wikimedia.org/T316118 (10Damilare) [15:49:23] fr-tech today's album recommendation is, In Search Of Sunrise 13.5: Amsterdam - disc 1 mix https://www.youtube.com/watch?v=PnqdtJfScAQ [15:58:56] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10donate.wikimedia.org: Broken links from Waystogive? - https://phabricator.wikimedia.org/T315268 (10jgleeson) a:03jgleeson [16:16:20] 10Fundraising-Backlog, 10fundraising-tech-ops: Deactivate fundraising accounts for eyener - https://phabricator.wikimedia.org/T316030 (10Jgreen) [16:33:59] 10Fundraising-Backlog, 10FR-WMF-Audit: Clarify audit processor logging - https://phabricator.wikimedia.org/T315034 (10Dwisehaupt) [16:45:03] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Ingenico form no longer supports iDEAL (was: Fatal exception of type "error") - https://phabricator.wikimedia.org/T315090 (10XenoRyet) 05Open→03Resolved [16:45:06] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Ingenico form no longer supports iDEAL (was: Fatal exception of type "error") - https://phabricator.wikimedia.org/T315090 (10XenoRyet) Evelyn has advised that there are no plans to use Ingenico for iDeal in the future, and it would n... [17:08:21] 10Fundraising-Backlog, 10fundraising-tech-ops: Deactivate fundraising accounts for eyener - https://phabricator.wikimedia.org/T316030 (10Jgreen) [17:08:50] 10Fundraising-Backlog, 10fundraising-tech-ops: Deactivate fundraising accounts for eyener - https://phabricator.wikimedia.org/T316030 (10Jgreen) [17:09:46] 10Fundraising-Backlog, 10fundraising-tech-ops: Deactivate fundraising accounts for eyener - https://phabricator.wikimedia.org/T316030 (10Jgreen) [18:10:08] 10Fundraising-Backlog, 10fundraising-tech-ops: Deactivate fundraising accounts for eyener - https://phabricator.wikimedia.org/T316030 (10Jgreen) [18:40:44] 10Fundraising-Backlog, 10fundraising-tech-ops: Deactivate fundraising accounts for eyener - https://phabricator.wikimedia.org/T316030 (10Jgreen) [18:49:07] 10Fundraising-Backlog, 10fundraising-tech-ops: Deactivate fundraising accounts for eyener - https://phabricator.wikimedia.org/T316030 (10Jgreen) [19:14:14] 10Fundraising-Backlog, 10fundraising-tech-ops: Deactivate fundraising accounts for eyener - https://phabricator.wikimedia.org/T316030 (10Jgreen) [19:25:32] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Proposal - restructure our repo to not use symlinks - https://phabricator.wikimedia.org/T258667 (10XenoRyet) 05Open→03Resolved [19:30:19] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fr-email-preference-center: New Custom Field - Opt Out Endowment Emails - https://phabricator.wikimedia.org/T248019 (10Eileenmcnaughton) 05Open→03Resolved a:03Eileenmcnaughton [19:30:59] 10Fundraising-Backlog, 10fundraising-tech-ops: Deactivate fundraising accounts for eyener - https://phabricator.wikimedia.org/T316030 (10Jgreen) [19:46:46] (03CR) 10Ejegg: "We'll need to maintain the .gitmodules file for the vendor submodule" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/825934 (https://phabricator.wikimedia.org/T314995) (owner: 10Eileen) [19:48:16] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/826369 [19:48:24] eileen___: I'm trying my own merge with a little change to the .gitmodules ^^^ [19:48:43] ejegg: yeah just saw your comment - duh [19:49:15] (03CR) 10CI reject: [V: 04-1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/826369 (owner: 10Ejegg) [19:49:21] hmm, still rejected [19:49:52] argh, it's a compser thing though: The .git directory is missing from /src/vendor/wikimedia/donation-interface [19:50:06] well that's annoying [19:50:31] ohhh, needs a vendor submodule update merged in too [19:50:53] ejegg: yeah - I went for the cherry-pick [19:51:03] eileen___: maybe let's do a merge of the lib updates previous to the civicrm change [19:51:10] I want to deploy that anyway [19:51:14] cool [19:51:18] then let's do the civi one as a regular merge [19:51:25] rocking, I'll set up the preliminary one [19:52:57] oh i see, you did the vendor update [19:53:05] I'll add that to my preliminary merge [19:53:12] just to simplify the follow-on one [19:53:33] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/826370 [19:53:35] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/826370 (owner: 10Ejegg) [19:53:51] (03CR) 10Ejegg: [V: 03+2 C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/826370 (owner: 10Ejegg) [19:54:16] ok, just gonna push that out right quick [19:55:40] !log civicrm upgraded from edfe2f16 to 632d5f5f [19:55:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:13:55] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/826377 [20:15:56] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/826377 (owner: 10Ejegg) [20:16:26] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/826377 (owner: 10Ejegg) [20:18:04] 10Fundraising-Backlog, 10fr-donorservices: Adyen duplicate donations after donors see error message - August 2022 - https://phabricator.wikimedia.org/T316157 (10MBeat33) [20:18:58] (03PS2) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/826369 [20:21:53] !log updated standalone SmashPig deploy from 13e9e9cc to 11ba0a1b [20:21:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:33:53] eileen___: looks like that got V+2 ! [20:34:08] yay - [20:35:07] 10Fundraising-Backlog, 10MW-1.39-notes (1.39.0-wmf.18; 2022-06-27): City field only on international Gpay checkout page - https://phabricator.wikimedia.org/T310307 (10XenoRyet) 05Open→03Resolved [20:58:10] 10Fundraising-Backlog, 10fundraising-tech-ops: Deactivate fundraising accounts for eyener - https://phabricator.wikimedia.org/T316030 (10Dwisehaupt) [21:04:43] ejegg: so just the vendor one merged so far? [21:06:51] eileen___: the whole merge is ready for a C+2: https://gerrit.wikimedia.org/r/826369 [21:07:40] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/826369 (owner: 10Ejegg) [21:07:45] gotta just do some kid stuff for a bit [21:08:28] (03Abandoned) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/825933 (owner: 10Eileen) [21:08:45] (03Abandoned) 10Eileen: Check in CiviCRM module [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/825934 (https://phabricator.wikimedia.org/T314995) (owner: 10Eileen) [21:10:24] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Proposal - restructure our repo to not use symlinks - https://phabricator.wikimedia.org/T258667 (10Eileenmcnaughton) woohoo [21:15:32] (03Abandoned) 10Eileen: Remove flush, move geocoders to post action [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809734 (owner: 10Eileen) [21:23:02] ok... so now we're getting a /different/ failmail for those recurring iDEAL donations! [21:23:09] baby steps [21:36:55] 10Fundraising-Backlog, 10fundraising-tech-ops: Deactivate fundraising accounts for eyener - https://phabricator.wikimedia.org/T316030 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt [21:39:11] 10Fundraising-Backlog, 10FR-Adyen, 10MediaWiki-extensions-DonationInterface: Orphan rectifier for Adyen - https://phabricator.wikimedia.org/T145708 (10Ejegg) [21:39:15] 10Fundraising Sprint NaN is a Number, 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog: Adyen - auto settle stopped prior donors (Pending transaction Resolver) - https://phabricator.wikimedia.org/T299692 (10Ejegg) [21:39:18] ejegg: I might do that deploy - it went ok on staging - just a refresh - but perhaps I turn off jobs [21:39:33] it is a larger r-sync - which could be a case for later in the day [21:39:54] but I'd also like to get it done [21:40:15] yep yep, seems fine to me [21:40:37] let's do it now-ish [21:40:54] I think all the long-running recurring jobs should be done [21:41:14] and if we need a kick from ops, at least dwisehaupt is still around [21:43:09] ? [21:43:27] yeah. i'm here to help if you need it. [21:44:03] heh, eileen___ has a deploy that may stress test the frdeploy tools [21:44:33] ok - I'll turn off queues & do it - deploy issue is just patch size [21:48:12] !log config revision changed from c2aa4158 to ab95bc89 [21:48:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:48:17] queues off [21:51:08] !log civicrm upgraded from 632d5f5f to ff9b377d [21:51:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:52:23] hmm - that went badly [21:52:24] ejegg: [21:53:06] which part went bad? [21:53:16] the failure of the site to load now [21:53:56] oh. yeah. [21:54:03] api calls work [21:54:21] hmmm... worth debugging or just rolling back? [21:54:25] hmm [21:54:29] yeah just deciding [21:54:33] let's peek at logs a sec? [21:55:06] Failed opening '/srv/org.wikimedia.civi [21:55:08] crm/civicrm/packages/Smarty/plugins/modifier.replace.php' for inclusion [21:55:15] doeok - it's improving - [21:55:30] cached templates? [21:56:06] looks like some CSS is missing, but the CRM content is there [21:56:35] let's look at paths [21:58:07] hmm, no way to load that part of the UI in this state [21:59:29] https://civicrm.wikimedia.org/civicrm/admin/setting/url?reset=1 [21:59:39] is that loading for you? [21:59:59] I get a blank center pane with a JS error [22:01:55] I got it I think [22:02:13] oh? [22:02:24] I didn't see anything likely in setting.get or vget [22:02:28] what did you do? [22:02:29] yeah - clearing caches from drupal seemed to cut it [22:02:52] nice! [22:02:57] yep, loading for me now too [22:03:12] yay - whew [22:03:24] one thing I don't know how to test is the email preference centre [22:03:57] oh, you should be able to use your own contact's link to change your preferred language [22:04:37] I just sent an opt-in request for my own contact [22:04:45] so then we run the preferences queue consumer [22:06:08] ok, I just ran it [22:06:19] and it seems to have run correctly [22:06:34] woot. [22:07:06] huh, might not have actually opted me in though [22:08:04] oh i see, there was a separate flag opting me out [22:08:22] just the silly confusion of separate privacy flags [22:08:44] hmm, maybe we should clear that other opt out flag when they affirmatively send an opt-in, though [22:12:09] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog: Custom field "AF_has_active_recurring_donation" is showing false positives - https://phabricator.wikimedia.org/T315102 (10Ejegg) This could have to do with the way we select which rows to export each night. We only export rows where th... [22:13:09] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog: Custom field "AF_has_active_recurring_donation" is showing false positives - https://phabricator.wikimedia.org/T315102 (10Ejegg) The number of recurring donors is low enough that we should be able to do a full-db search for ppl with sup... [22:13:15] separate thing - now we are not breeding doctors there is a cleanup https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/824334 [22:13:29] ejegg: ug - so that goes to triggers? [22:13:50] eileen: oh, that ticket? [22:14:14] no, I think it might have to do with us not getting the 'subscription end' notification or something [22:14:23] oh - your latest updates [22:14:35] dog needs a trip outside. i'll have my phone with in case anything goes wrong. i'll just be around the block. [22:14:51] so we just stop getting payments, but no signal to update the status of the contribution_recur row [22:14:59] actually I might know - I think there is antother ticket - we don't update contact.modified_date when we modify recurrings I think [22:15:03] I'm guessing those CIDs are mostly paypal [22:15:41] ohh, we would want to update that modified date on recurring updates too, for sure [22:16:32] aha - https://phabricator.wikimedia.org/T293945 [22:17:40] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog: Custom field "AF_has_active_recurring_donation" is showing false positives - https://phabricator.wikimedia.org/T315102 (10Eileenmcnaughton) May be a symptom of https://phabricator.wikimedia.org/T293945 [22:19:10] eileen: so that would help with some of these ppl, but for the ones where we have a recurring donation that is still 'active' in civi but hasn't had a payment for years, we'd need a different fix [22:19:42] ah ok - what about a sched job that changed the status after x long [22:19:54] yep, that could work [22:20:07] let's see if they are all super-old [22:20:13] if so, maybe just a one-time data fix [22:23:17] (03CR) 10Ejegg: [C: 03+2] IDE cheer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/824329 (owner: 10Eileen) [22:39:37] back [22:40:58] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog: Custom field "AF_has_active_recurring_donation" is showing false positives - https://phabricator.wikimedia.org/T315102 (10Ejegg) 936002 - 'active' paypal recurring but no donation in a while 75603 - 'active' paypal recurring but no dona... [22:44:24] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog: Custom field "AF_has_active_recurring_donation" is showing false positives - https://phabricator.wikimedia.org/T315102 (10Ejegg) 1683114 shows the recurrings were both cancelled on Jan 3rd, and their Acoustic record shows it was last mo... [22:49:43] 10Fundraising-Backlog: Remaining campaign markets need Adyen set as default - https://phabricator.wikimedia.org/T316168 (10EMartin) [22:50:22] ohhh, AF_has_active is one of the triggers... I thought it was just calculated on the fly in the export [22:50:25] derp [22:50:29] why did I assume that? [22:50:51] ok, so let's make that trigger on the contribution_recur table [22:52:43] 10Fundraising-Backlog, 10FR-Docker, 10Epic: Epic: feature parity with vagrant on docker - https://phabricator.wikimedia.org/T262971 (10Ejegg) [22:52:45] 10Fundraising Sprint Corrugated super slide, 10Fundraising Sprint Esperantoland, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: De-symlink our repos - https://phabricator.wikimedia.org/T273822 (10Ejegg) 05Open→03Resolved [22:53:27] (03Merged) 10jenkins-bot: IDE cheer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/824329 (owner: 10Eileen) [22:57:33] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Alerts: INVALID_MESSAGE Recurring donation, but no subscription ID or recurring payment token found failmail coming from the listener on ideal - https://phabricator.wikimedia.org/T315031 (10Ejegg) Good news is we're getting the tokens now! Bad new... [22:59:32] 10Fundraising-Backlog, 10FR-WMF-Audit: Clarify audit processor logging - https://phabricator.wikimedia.org/T315034 (10Ejegg) To clarify this ticket: this is just for the wmf_audit module (or wherever it gets moved in the Civi extension), not for the python PayPal audit parser [23:00:29] ah never mind, the has_recurring IS calculated on the export [23:01:22] so we could also update the join there and make it check that there has been a contribution in the last 45 days [23:11:24] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog: Custom field "AF_has_active_recurring_donation" is showing false positives - https://phabricator.wikimedia.org/T315102 (10Ejegg) In the calculated table we use to keep track of who has an active recurring, 1683114 is correctly indicated... [23:12:02] queues are still stopped, yes? [23:13:23] oh yes - fixing [23:13:35] cool. just wanted to verify. :) [23:15:15] lol I don't suppose there is enough backed up to squeeze in a performance fix /test [23:16:09] if would be a very short test. :) [23:16:14] :-) [23:16:20] it would be that is. only about 2k in the queue. [23:20:36] ejegg: what do you think - worth trying https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/civicrm/+/820400 ? [23:20:44] (I need to re-do on crm) [23:21:39] oh, for sure! [23:22:02] ok gimme a couple of mins to update the patch [23:23:23] (03PS1) 10Eileen: Fix another cache miss [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/826402 [23:23:29] ok ejegg ^^ [23:24:06] (03Abandoned) 10Eileen: Fix another cache miss [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/820400 (owner: 10Eileen) [23:26:53] (03PS2) 10Eileen: Fix another cache miss [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/826402 [23:28:18] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 5702 5000, payments-init is 2080 2000, pending is 1668 1500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 100 days 9 hours - memory use is 14.48M (peak 121.65M, 0.26% of max, fragmentation 1.48%), connected_slaves is 3, donations is 2003, jobs is 0, jobs-adyen is 1951, jobs-paypal is 904, recurring is 37, refund is 0, unsubscribe i [23:28:18] s://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [23:29:00] eileen looks like you are ignoring the $force param there? Maybe mark it deprecated or something if you don't want it to be used [23:30:22] ejegg: yeah looks like the upstream drops it - https://github.com/civicrm/civicrm-core/pull/24297/files [23:31:43] fwiw, i acked the redis alert in the paging software but not icinga since we are going to reenable the queues soon. [23:32:04] cool [23:32:25] ejegg: I think we can ignore that - the fix upstream is better but the existing won't hurt [23:32:53] oh - this is the right one https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/826402 [23:33:18] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 5959 5000, payments-init is 2177 2000, pending is 1743 1500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 100 days 9 hours - memory use is 15.00M (peak 121.65M, 0.27% of max, fragmentation 1.46%), connected_slaves is 3, donations is 2094, jobs is 0, jobs-adyen is 2057, jobs-paypal is 947, recurring is 40, refund is 0, unsubscribe i [23:33:18] s://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [23:34:31] 10Fundraising-Backlog: Chargebacks coming in for no authorization - https://phabricator.wikimedia.org/T316173 (10EMartin) [23:38:22] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 6230 5000, payments-init is 2275 2000, pending is 1817 1500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 100 days 9 hours - memory use is 15.67M (peak 121.65M, 0.27% of max, fragmentation 1.43%), connected_slaves is 3, donations is 2185, jobs is 0, jobs-adyen is 2122, jobs-paypal is 992, recurring is 40, refund is 0, unsubscribe i [23:38:22] s://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [23:40:07] actually, i'll ack in icinga too since it's spammy with how long the message is in irc. [23:41:12] ACKNOWLEDGEMENT - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 6230 5000, payments-init is 2275 2000, pending is 1817 1500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 100 days 9 hours - memory use is 15.67M (peak 121.65M, 0.27% of max, fragmentation 1.43%), connected_slaves is 3, donations is 2185, jobs is 0, jobs-adyen is 2122, jobs-paypal is 992, recurring is 40, refund is 0, unsub [23:41:12] 27 Dwisehaupt known - queues stopped on purpose. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [23:47:11] ejegg: are you looking at the patch now? [23:51:49] hi fr-tech, is there something causing the queue to slow? I see in superset it's 124 minutes. I'm not panicking but just curious if it's normal or something else [23:53:37] haley_: yep - it's off at the moment - it was off for a deploy & then I forgot to turn it back on straight away & then we thought we might try a perf test on a patch [23:54:28] gotcha, thanks eileen! I picked a bad time to put up a new test haha. No worries though, I'll check back later [23:55:02] haley_: we can turn it back on - ejegg was gonna review the test but he might be parenting so if he isn't back soonish will turn it on anyway [23:55:51] sounds good! no rush on my part [23:58:02] cool