[00:00:48] the threshold should be 250 in 5 mins [00:00:57] I'm wondering if the 5 mins should be 7 mins [00:01:54] hmm - prob also checking for new contacts is part of the prob - shall I try 7 mins? [00:04:14] (03Merged) 10jenkins-bot: Add fastArrayDecorator to mixinBoot & mixinScan caches [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/856703 (https://phabricator.wikimedia.org/T322172) (owner: 10Eileen) [00:05:40] hmm eileen is that graph yellow because you made it yellow [00:05:43] or what does the yellow vs green made [00:06:11] I didn't do anything on purpose & I think it already way [00:06:13] was [00:10:52] eileen: it doesnt look like it was following its normal schedule though [00:13:47] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Review our queue speed - is it OK? - https://phabricator.wikimedia.org/T322791 (10greg) No thank-you queue bump today during the high-volume test. [00:14:00] 10Fundraising-Backlog, 10FR-Japan, 10fr-donorservices: Japan Form Variations for Testing for Q3 - https://phabricator.wikimedia.org/T322793 (10TSkaff) This [[ https://docs.google.com/presentation/d/1d1QkcJTj5MEwE201BnJ03DpdL4_zefoT7joLBAOnE0Q/edit#slide=id.g17b4400b0ff_0_1 | visualization ]] might spell it o... [00:18:47] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Review our queue speed - is it OK? - https://phabricator.wikimedia.org/T322791 (10Eileenmcnaughton) We just had a 1 hour test run - observations The overall pattern was "around 6 donations per second, disrupted by a peri... [00:23:16] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Oh Behave Dedupe - https://phabricator.wikimedia.org/T323081 (10Eileenmcnaughton) [00:24:17] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Oh Behave Dedupe - https://phabricator.wikimedia.org/T323081 (10Eileenmcnaughton) [00:24:47] cstone: wasn't it? https://phabricator.wikimedia.org/T323081 - I created a new phab for the deduper kicking in ^^ [00:26:12] time math melts my brain but every hour from 0 to 23 should have had it starting at 0 and not :25 at 22, but what i think its 22 is maybe it thinks is a different time [00:28:47] oh I thought it started at 25 from my read of the graph [00:30:52] yeah but from the cron shouldn't it have started at 0 [00:30:57] entirely possible im reading it wrong too [00:31:22] i was trying to separate out just the dedupe on that graph and failing [00:53:25] so expanding the timeframe [00:53:26] https://frmon.wikimedia.org/d/lIN1akDVz/process-control-precise?orgId=1&from=1668463140000&to=1668470399000 [00:53:42] it ran at 22.25 & 22.50 [00:53:45] i got this figured out too not sure if its helpful to anyone but me https://frmon.wikimedia.org/d/2-uC2WkVz/christine-test?orgId=1&from=now-6h&to=now [00:53:52] why is it running at 50? [00:54:23] Every 25 minutes, at 12:00 AM through 01:59 AM and 05:00 AM through 11:59 PM [00:54:43] ahha I pasted it into crontab guru and it still confused me [00:54:44] ok [00:56:52] should it only run at 25 and 50 for two hours though or am I reading that wrong too? [00:59:15] yeah - but I'm looking back further & still a bit confused - ah it 'resets' the 'every' each hour - these all ran but one [00:59:16] 20:25 [00:59:16] 20:50 [00:59:16] 21:00 [00:59:16] 21:25 [00:59:16] 21:50 [00:59:17] 22:00 [00:59:17] 22:25 [00:59:18] 22:50 [00:59:18] 23:00 - still running the last one so didn't run [00:59:19] 23:25 [00:59:19] 23:50 [01:06:25] ok crontabguru was failing me i understand now haha [01:07:04] so doesnt run at all 3-5 and then :00, :25, :50 otherwise? [01:10:21] its not running some of those [01:11:53] yeah - so it ran in all cases - but the cron should exit early if the server is busy - which it didn't [01:12:12] and did the opposite and ran too long [01:17:27] (03CR) 10Cstone: CreateRecurringPaymentsProfile (032 comments) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) (owner: 10Wfan) [01:17:46] im gona head out eileen , have a good rest of your day! [01:17:57] night [01:19:30] (03CR) 10Eileen: [C: 03+2] "Yep - worked in my testing - yay" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/856688 (https://phabricator.wikimedia.org/T321251) (owner: 10Eileen) [01:19:59] (03CR) 10Eileen: [C: 03+2] Add Clean Unused language Options api (032 comments) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/856688 (https://phabricator.wikimedia.org/T321251) (owner: 10Eileen) [01:20:05] ah dang, we should make the proxy forwarder add the X-FORWARDED-FOR header [01:30:23] AndyRussG: I want to test with Redis locally - I see it is in our docker but [01:30:24] redis-cli [01:30:24] Could not connect to Redis at 127.0.0.1:6379: Connection refused [01:30:40] so I'm not sure what I need to know to use it [01:31:13] I guess it is just used for Queues? [01:31:37] (03Merged) 10jenkins-bot: Add Clean Unused language Options api [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/856688 (https://phabricator.wikimedia.org/T321251) (owner: 10Eileen) [01:32:14] eileen: mostly just for queues, yep [01:32:17] and a little bit of caching [01:32:28] plus the sequence generator for contribution_tracking [01:32:33] Hmm I guess I was running redis-cli locally so that must be my local - just trying to wire local civi up to it [01:32:54] does it have a password? [01:33:06] eileen, in the fr-dev root directory there is a script queues-redis-cli.sh [01:33:39] let's see, looks like no password [01:34:39] so the connection info is also in the config/smashpig/main.yaml file [01:34:47] the internal hostname is 'queues' [01:34:53] port 6379 [01:34:54] tcp [01:36:13] ohhh i see, you're trying to wire up your non-docker civi [01:36:18] sorry, misunderstood for a bit [01:36:37] let's see if it exposes a port [01:37:11] darn, not seeing one [01:37:39] you could probably make it expose one by adding a ports: section to the queues: config in docker-compose.yml [01:37:48] then just restart the containers [01:37:59] (think it doesn't even need a rebuild) [01:40:27] ejegg: no docker civi - I think I might be needing to use that internal hostname somehow [01:40:54] yep, it should be just 'queues' and docker-dns will figure it out [01:42:13] ok, let me look at the prod config to remind myself the format [01:42:22] it's in civicrm.settings.php, right? [01:42:22] hmm - so you are saying drupal is already using it? (not quite sure how I specify the host name in civi but trying) [01:43:00] so it's currently being hit from the civicrm box, but afaik only by the queues code which runs through smashpig functions [01:43:34] ok - so it must be possible :-) [01:44:36] ok, probably just using define('CIVICRM_DB_CACHE_HOST', 'queues') should do it [01:44:45] along with define('CIVICRM_DB_CACHE_PORT', 6379); [01:44:50] and define('CIVICRM_DB_CACHE_CLASS', 'Redis'); [01:45:13] no password needed - that key should be optional for the Civi Redis cache class, right? [01:46:21] it did! [01:46:25] yay! [01:46:33] I wonder how we can add that to our image [01:47:05] I guess that wouldn't go in the wmff build, as it's docker-specific [01:47:22] we have some other docker-specific bits though, don't we? In another file I think? [01:48:59] (03PS1) 10Eileen: Add Redis to our civi instance [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/856715 [01:49:12] that isn't working yet ^^ - but something like it [01:50:50] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Oh Behave Dedupe - https://phabricator.wikimedia.org/T323081 (10Ejegg) Looks like log_civicrm_contribution might be a useful proxy? select count(*) from log_civicrm_contribution where log_date > date_sub(now(), interval... [01:53:16] nice eileen, just needs that CIVICRM_DB_CACHE_HOST to be defined to 'queues', I think? [01:53:39] ejegg: yeah - it is locally - but the problem is that location is not being loaded early enough [01:53:45] oic [01:54:37] it works if I do [01:54:38] mv /srv/config/exposed/civicrm/settings.d/130-redis.php /srv/civi-sites/wmff/drupal/sites/default/civicrm.settings.d/pre.d/ [01:55:05] is that location only loaded by local - I think it might be [01:56:44] OK - it is definitely not loaded by live - I'm less sure about CI though [01:57:56] (03PS1) 10Eileen: Add Redis to CiviCRM on local dev [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/856716 [01:58:03] 10Fundraising-Backlog, 10FR-Japan, 10fr-donorservices: Japan Form Variations for Testing for Q3 - https://phabricator.wikimedia.org/T322793 (10Ejegg) Hi @TSkaff somehow I'm only seeing Variation 1 at that link. It sounds like version / variation 2 is adding another set of inputs? > Under the existing name f... [01:59:53] (03CR) 10CI reject: [V: 04-1] Add Redis to CiviCRM on local dev [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/856716 (owner: 10Eileen) [02:00:05] ok - that's not a big surprise.... [02:01:43] ejegg: any idea of a conditional we could add that would distinguish between our docker & CI [02:03:07] (03CR) 10Wfan: CreateRecurringPaymentsProfile (032 comments) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) (owner: 10Wfan) [02:03:16] 10Fundraising-Backlog, 10FR-Japan, 10fr-donorservices: Japan Form Variations for Testing for Q3 - https://phabricator.wikimedia.org/T322793 (10Ejegg) Sorry, now GDocs is loading all the slides for me, as well as the note under version 2 with the rationale for the extra fields even if they aren't saved to Civ... [02:04:55] (03PS2) 10Eileen: Add Redis to CiviCRM on local dev [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/856716 [02:04:58] ok trying ^^ [02:23:10] ahhh silly me, i've had the proxy params all wrong for the remote forwarding box [02:23:42] been fumbling around trying to figure out why i can't test geoip that way [02:54:16] (03PS1) 10Eileen: Backport Redis Cache fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/856720 (https://phabricator.wikimedia.org/T322172) [03:05:35] darn it, now it has the headers but the forwarded-for address is still a 172.16 (private) address [03:06:33] maybe that's the SSL terminator [03:35:02] (03PS2) 10Ejegg: Update minFraud and GeoIP2 libraries [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/854872 (https://phabricator.wikimedia.org/T281443) (owner: 10AndyRussG) [03:35:37] (03CR) 10Ejegg: [C: 03+2] "Thanks! Minfraud and GeoIP still seem to work fine with no code changes." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/854872 (https://phabricator.wikimedia.org/T281443) (owner: 10AndyRussG) [03:36:07] thx ejegg :) [03:36:54] I had to hack the code a bit to provide a non-internal IP address to the geoip... [03:38:11] I'm wondering how to see if the wmcloud SSL terminator is providing any of its own forwarded-for headers - I thought that nginx would have been passing them through unchanged [03:38:26] (03Merged) 10jenkins-bot: Update minFraud and GeoIP2 libraries [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/854872 (https://phabricator.wikimedia.org/T281443) (owner: 10AndyRussG) [03:38:45] but it doesn't seem to give me any x-forwarded-for when i remove the 'include proxy_params' bit [03:40:34] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Patch-For-Review: Excessive Redis cache hits - https://phabricator.wikimedia.org/T322172 (10Eileenmcnaughton) Not sure this got linked in https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/856703 [03:41:21] (03Abandoned) 10Eileen: Add Redis to our civi instance [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/856715 (owner: 10Eileen) [03:41:40] 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface: Log Minfraud ID for each minfraud request - https://phabricator.wikimedia.org/T323072 (10Ejegg) 05Open→03Invalid Oops, this is already happening. It's just inside a JSON blob in the payments.fraud log, rather than the processor-specific logs.... [03:45:00] oh hmm, there's a smashpig library update that'd go along with that. [03:45:06] well, it's late for a deploy anyway [03:45:30] guess I'll head off to bed and do it in the morning. See ya! [05:17:56] 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10JavaScript, 10ci-test-error (WMF-deployed Build Failure): flaky qunit test in CentralNotice: Uncaught TypeError: Cannot read property 'timeRemaining' of undefined - https://phabricator.wikimedia.org/T318103 (10AndyRussG) [05:40:21] (03CR) 10AndyRussG: "thx for this! (smallest of nit-picks inline...)" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/854057 (https://phabricator.wikimedia.org/T318867) (owner: 10Jgleeson) [05:51:52] (03CR) 10AndyRussG: [C: 03+2] "Thx so much for this!!!! :)" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/854581 (https://phabricator.wikimedia.org/T318756) (owner: 10Jgleeson) [12:39:23] 10Fundraising-Backlog, 10FR-Japan, 10fr-donorservices: Japan Form Variations for Testing for Q3 - https://phabricator.wikimedia.org/T322793 (10CDenes_WMF) Thanks Elliott! Yes the slides are much easier to digest IMO :) (wikilove to Thea for creating them). We could either *not* store the Furigana at all; or... [12:47:50] 10Fundraising-Backlog, 10Content-Transform-Team-WIP, 10Patch-For-Review, 10Wikipedia-Android-App-Backlog (Android Release FY2022-23): 2022 English fundraising messages in Android app - https://phabricator.wikimedia.org/T322951 (10Jgiannelos) [12:48:22] 10Wikimedia-Fundraising-Banners, 10Content-Transform-Team-WIP, 10Wikipedia-iOS-App-Backlog, 10Patch-For-Review, 10ios-app-v7.0.1: 2022 English fundraising messages in iOS app - https://phabricator.wikimedia.org/T322950 (10Jgiannelos) [14:07:14] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (BraintreeApi:5, BraintreeGateway:5) 10 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [14:12:14] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [14:23:30] (03CR) 10Jgleeson: Move GetExpressCheckoutDetails mapping into API (031 comment) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/854057 (https://phabricator.wikimedia.org/T318867) (owner: 10Jgleeson) [14:24:13] (03PS5) 10Jgleeson: Move GetExpressCheckoutDetails mapping into API [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/854057 (https://phabricator.wikimedia.org/T318867) [14:25:16] (03PS12) 10Jgleeson: Update PayPal EC approvePayment/doExpressCheckoutPayment status mapping. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/854581 (https://phabricator.wikimedia.org/T318756) [14:25:29] (03PS8) 10Jgleeson: Add Paypal EC PaymentProvider Tests [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/855749 (https://phabricator.wikimedia.org/T318756) [14:25:57] (03PS6) 10Jgleeson: Update Adyen and Ingenico to use IGetLatestPaymentStatusProvider. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/856617 [14:26:17] good spot on the cmt message AndyRussG! thanks for the review [14:26:22] back soon [16:22:36] (03CR) 10Cstone: [C: 03+2] Backport Redis Cache fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/856720 (https://phabricator.wikimedia.org/T322172) (owner: 10Eileen) [16:31:28] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/857008 [16:31:34] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/857008 (owner: 10Ejegg) [16:32:40] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/857008 (owner: 10Ejegg) [16:35:05] (03PS1) 10Ejegg: Update DonationInterface and vendor [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/857011 [16:35:08] (03CR) 10Ejegg: [C: 03+2] Update DonationInterface and vendor [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/857011 (owner: 10Ejegg) [16:35:59] (03Merged) 10jenkins-bot: Backport Redis Cache fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/856720 (https://phabricator.wikimedia.org/T322172) (owner: 10Eileen) [16:45:05] (03Merged) 10jenkins-bot: Update DonationInterface and vendor [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/857011 (owner: 10Ejegg) [16:52:02] 10Fundraising-Backlog, 10MediaWiki-Core-Skin-Architecture, 10MediaWiki-extensions-CentralNotice, 10Technical-Debt: [Maintenance request] CentralNotice should not use SkinTemplateNavigation hook to add special page tabs - https://phabricator.wikimedia.org/T315562 (10Jdlrobson) [17:09:54] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10User-greg: Refill minfraud request buffer - https://phabricator.wikimedia.org/T323069 (10greg) [18:14:48] 10Fundraising-Backlog: Document Data Delivery Tool - https://phabricator.wikimedia.org/T323144 (10greg) [18:27:56] 10Fundraising-Backlog: Document Data Delivery Tool - https://phabricator.wikimedia.org/T323144 (10greg) https://fundraising.frdev.wikimedia.org:444/donor_data/ (thanks dallas!) [18:54:29] (03CR) 10Cstone: CreateRecurringPaymentsProfile (031 comment) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) (owner: 10Wfan) [19:05:43] (03PS8) 10Wfan: CreateRecurringPaymentsProfile [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) [19:06:13] (03CR) 10CI reject: [V: 04-1] CreateRecurringPaymentsProfile [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) (owner: 10Wfan) [19:07:16] (03PS9) 10Wfan: CreateRecurringPaymentsProfile [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) [19:10:33] (03CR) 10AndyRussG: [C: 03+2] "Thanks!!!!!!!!! :)" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/854057 (https://phabricator.wikimedia.org/T318867) (owner: 10Jgleeson) [19:11:07] (03CR) 10Wfan: CreateRecurringPaymentsProfile (031 comment) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) (owner: 10Wfan) [19:11:11] (03Merged) 10jenkins-bot: Move GetExpressCheckoutDetails mapping into API [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/854057 (https://phabricator.wikimedia.org/T318867) (owner: 10Jgleeson) [19:11:14] (03Merged) 10jenkins-bot: Update PayPal EC approvePayment/doExpressCheckoutPayment status mapping. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/854581 (https://phabricator.wikimedia.org/T318756) (owner: 10Jgleeson) [19:25:07] 10Wikimedia-Fundraising-Banners, 10Content-Transform-Team-WIP, 10Wikipedia-iOS-App-Backlog, 10ios-app-v7.0: 2022 English fundraising messages in iOS app - https://phabricator.wikimedia.org/T322950 (10LGoto) [19:25:48] thanks for the review AndyRussG! [19:28:19] (03CR) 10Cstone: CreateRecurringPaymentsProfile (032 comments) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) (owner: 10Wfan) [19:34:04] (03PS1) 10Eileen: Add fastArrayDecorator to mixinBoot & mixinScan caches [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/857036 (https://phabricator.wikimedia.org/T322172) [19:35:14] (03CR) 10Eileen: [C: 03+2] "Cherry picking this in to do just this & then the other" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/857036 (https://phabricator.wikimedia.org/T322172) (owner: 10Eileen) [19:38:15] !log payments-wiki upgraded from a058fdbc to bba997aa [19:38:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:38:32] !log turned off CiviCRM dedupe jobs for queue speed measurements [19:38:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:42:25] (03CR) 10Jgleeson: [C: 04-1] "Thanks for this. Could we rejig the tests slightly?" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) (owner: 10Wfan) [19:51:50] 10Wikimedia-Fundraising-Banners, 10Content-Transform-Team-WIP, 10Wikipedia-iOS-App-Backlog, 10ios-app-v7.0: 2022 English fundraising messages in iOS app - https://phabricator.wikimedia.org/T322950 (10Tsevener) @Jgiannelos Can we get this changed deployed by the end of this week? We've done some light local... [20:13:19] !log civicrm upgraded from 3eba6ad3 to 16167e9a [20:13:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:14:22] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/857042 [20:14:40] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/857042 (owner: 10Eileen) [20:32:40] 10Wikimedia-Fundraising-CiviCRM: Donations queue consumer should write stats every minute - https://phabricator.wikimedia.org/T323162 (10Ejegg) [20:33:49] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Excessive Redis cache hits - https://phabricator.wikimedia.org/T322172 (10Eileenmcnaughton) a:03Eileenmcnaughton [20:34:27] 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Decide a naming convention for our extensions & document on wikitech - https://phabricator.wikimedia.org/T321638 (10Eilee... [20:42:43] !log civicrm upgraded from 16167e9a to d85589e8 [20:42:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:44:28] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Donations queue consumer should write stats every minute - https://phabricator.wikimedia.org/T323162 (10greg) [20:52:58] 10Fundraising-Backlog: Unsubscribe link not communicating with Civi as it used to - https://phabricator.wikimedia.org/T322084 (10greg) [21:01:09] 10Fundraising Tech - Chaos Crew, 10MediaWiki-extensions-CentralNotice, 10MediaWiki-extensions-Translate: Can’t mark CentralNotice translations as reviewed - https://phabricator.wikimedia.org/T308614 (10Damilare) p:05Triage→03Low [21:01:17] 10Fundraising Sprint Undefined, 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Civi-Dedupe, and 2 others: Civi Dedupe: show Last Donated date at top of legacy dedupe screen - https://phabricator.wikimedia.org/T256314 (10Eileenmcnaughton) [21:01:37] 10Fundraising Sprint Undefined, 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Civi-Dedupe, and 2 others: Civi Dedupe: show Last Donated date at top of legacy dedupe screen - https://phabricator.wikimedia.org/T256314 (10Eileenmcnaughton) @shust all good now? [21:02:07] 10Fundraising Sprint Undefined, 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Civi-Dedupe, and 2 others: Civi Dedupe: show Last Donated date at top of legacy dedupe screen - https://phabricator.wikimedia.org/T256314 (10greg) p:05Triage→03Medium [21:02:23] 10Fundraising Tech - Chaos Crew: Fix sql error when calculating first donation date where a $0 or negative donation is involved - https://phabricator.wikimedia.org/T322189 (10Eileenmcnaughton) 05Open→03Resolved [21:02:25] 10Fundraising Tech - Chaos Crew: Differences between civicrm.wmf_donor table and Civi UI - https://phabricator.wikimedia.org/T321920 (10Eileenmcnaughton) [21:02:46] 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising Sprint Undefined, 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Clean up our five character locale data.... - https://phabricator.wikimedia.org/T321251... [21:02:55] 10Fundraising Tech - Chaos Crew: Write unit tests for Damaged Extension - https://phabricator.wikimedia.org/T322640 (10greg) p:05Triage→03Low [21:26:08] (03PS1) 10Ejegg: Query different table for queue backup [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857072 (https://phabricator.wikimedia.org/T323081) [21:32:32] eileen: I need to modify the clean language function to only look at the is_deleted = 0 contacts, otherwise no language will be cleaned, we got so many deleted contact that is using the old invalid language local [21:40:31] (03PS1) 10Ejegg: Export donations qc stats every minute [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857074 (https://phabricator.wikimedia.org/T323162) [21:41:05] jgleeson: looks like maybe that will work to export the stats every minute ^^^ [21:41:14] I just need to smoke test a bit [21:41:38] music recommendation for right now: https://crescentcitysounds.org/albums/young-fellaz-brass-band-block-party [21:42:29] The coolest part is that it's from a public streaming music site run by the New Orleans public library [21:44:20] 10Fundraising-Analysis, 10Fundraising-Backlog, 10FR-Email: Diff Tool to compare two emails - Test vs Control - https://phabricator.wikimedia.org/T322891 (10greg) Is this a request for a tool within Acoustic to compare two emails/drafts? [21:46:31] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10fr-donorservices: Civi: Forget Me button yielding parameters errors - https://phabricator.wikimedia.org/T322796 (10MBeat33) Thanks for keeping this in Undefined Sprint, @greg If needed we can also push it to Q3. [21:46:35] looking ejegg [21:47:09] jgleeson: oh sorry, it doesn't actually work I just realized [21:47:24] that timerDiff only exists after the end, I think [21:47:50] so I have to find a better way to get the running timer without stopping it [21:49:28] I guess I can update the DonationStatsCollector [21:49:47] to give it a getRunningTimer($namespace) function [21:49:51] I think we can do that without hacking it [21:50:26] just on the arch call but will take a look after it and see if anything comes to mind [21:50:34] oh sorry, carry on [21:50:44] this is totally low priority [21:50:49] I also think we could get some useful info by including the batch time in that graph [21:51:04] as we already post that to prometheus [21:51:58] maybe... but I'm also thinking maybe we should have it sleep a sec when the queue is empty so the batch time is always padded out to the full limit [21:52:33] we already do that with one of the other QCs - meybe pending or contribution tracking? [21:52:40] (03CR) 10CI reject: [V: 04-1] Export donations qc stats every minute [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857074 (https://phabricator.wikimedia.org/T323162) (owner: 10Ejegg) [21:53:09] hmm [21:53:46] is the padding to make sure each run is the same length? [21:54:04] i.e. same batch time [21:54:24] I was thinking shorter batch times vs longer might show useful patterns [21:54:37] the obvious one being performance degrading with time [21:54:48] or maybe improving due to cache? who knows [21:58:01] the sleep is to make sure we're pretty continuously consuming [21:58:37] as opposed to getting to 0 after 40 sec, then ending the run and not running again for another 2 min 20 sec while donations build up in the queue [22:00:32] 10Fundraising Tech - Chaos Crew: Adyen: Client side error on Safari even with successful donation - https://phabricator.wikimedia.org/T310737 (10greg) p:05Triage→03Low [22:01:20] 10Fundraising Tech - Chaos Crew: dLocal Safari internal errors on redirect - https://phabricator.wikimedia.org/T318744 (10greg) p:05Triage→03Medium [22:01:23] 10Fundraising Tech - Chaos Crew: dLocal Insecure Form warnings - https://phabricator.wikimedia.org/T318742 (10greg) p:05Triage→03Medium [22:01:54] 10Fundraising Tech - Chaos Crew: Adyen audit sending not refunded recurring ideal donations to the refund queue - https://phabricator.wikimedia.org/T320286 (10greg) p:05Triage→03Medium [22:02:31] 10Fundraising Tech - Chaos Crew: Civi DB Error: constraint violation detected from civicrm_mailing_event_delivered from recurring_queue_consume log - https://phabricator.wikimedia.org/T319445 (10greg) [22:03:17] 10Fundraising Tech - Chaos Crew: Civi DB Error: constraint violation detected from civicrm_mailing_event_delivered from recurring_queue_consume log - https://phabricator.wikimedia.org/T319445 (10greg) p:05Triage→03Medium [22:03:34] 10Fundraising Tech - Chaos Crew: php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution - https://phabricator.wikimedia.org/T318956 (10greg) p:05Triage→03Low [22:03:47] ok so you wanna make it output every minute [22:03:54] 10Fundraising Tech - Chaos Crew: Intermittent Apple Pay timeouts when testing locally - https://phabricator.wikimedia.org/T318808 (10greg) p:05Triage→03Low [22:04:07] what frequency does the prom consumer run at [22:04:19] otherwise I think you might just overwrite the files each time [22:04:41] that would be the first question for dwisehaupt or Jeff_Green [22:04:57] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Alerts: Update the frequency of smart-group cache refresh - https://phabricator.wikimedia.org/T321337 (10greg) p:05Triage→03Medium [22:05:25] I guess if we can get it to scan more frequently we could just start changing the filenames [22:05:34] I think atm it's a pretty constant filename [22:05:38] 10Fundraising Tech - Chaos Crew: Broken links from Waystogive? - https://phabricator.wikimedia.org/T315268 (10greg) p:05Triage→03Low [22:05:47] let's see on civi1001 [22:06:02] afaik the prom consumer is a glorified/stripped down webserver, all it does is serve the contents of the prom files when the prometheus scrapes metrics [22:06:08] 10Fundraising Tech - Chaos Crew: Fail Mail (civi1001) run-job: Fetch CiviMail Bounces failed with code - https://phabricator.wikimedia.org/T323057 (10greg) p:05Triage→03Medium [22:06:17] so whatever file is there will be scraped on the regular prometheus 1 minute interval [22:06:48] we'd need them to line up which might be tricky [22:07:00] 10Fundraising Tech - Chaos Crew: Adyen audit sending not refunded recurring ideal donations to the refund queue - https://phabricator.wikimedia.org/T320286 (10greg) p:05Medium→03High [22:07:01] the exporter and the prometheus consumer [22:07:09] prometheus can't do that [22:07:59] if you need anything better than sloppy approximate times +/- a minute or two, prometheus can't do it [22:08:07] (sorry about the phab spam) [22:09:39] ok. I wonder if we can do this by refining our metrics [22:09:44] just looking now [22:11:26] I started experimenting with parallel collection of prometheus metrics into mysql, going that route we can ingest with precise per-metric timestamps, but it's harder to work with as a data source in grafana [22:11:43] hmm no I guess that will still limit us to only producing new metrics every 3 minutes [22:12:31] ok so ejegg's idea of calling export on a timer feels in the right area then [22:12:54] what's the 3 minute limit? [22:16:17] so at the moment the queue consumer runs every 3 minutes I think [22:16:35] and the queue consumer stats tool which generates the prom status runs inside that process [22:16:51] the issue at the moment is that ejegg wants to see new data quicker than 3-min intervals [22:17:03] which data? [22:17:22] the donation processing rates for now [22:17:32] * Jeff_Green goes reading backscroll [22:17:59] 10Fundraising Tech - Chaos Crew: Name changes required for Ingenico 3DS2.0 - https://phabricator.wikimedia.org/T312877 (10EMartin) Pinged Ingenico today 15 Nov for an update. This was to be ready in November. [22:18:00] we have most of the queues in prometheus already [22:19:01] seems like the simplest would be to get whatever queue metrics are needed into prometheus and do whatever math is needed in the graph query [22:19:46] Jeff_Green: do you mean recording the activity at the redis level? [22:19:56] yeah [22:20:35] hmm so that would show us how quickly messages leave the queue but it might not give us the full picture of how fast they are processed [22:20:44] wfan - just saw ^^ - yeah put up a patch [22:21:01] like it'll show us how many troops jump from the plane but not how fast they fall and if they hit ducks along the way [22:21:37] jgleeson: I'm not sure if redis can report inserts to the queues... [22:22:26] I think it's the actual consumer processing we're interesting in knowing more about [22:22:38] in the hope that we can optimise [22:22:42] the php code that is [22:22:52] and also identify bottlenecks [22:23:05] we have something trying to do that already [22:23:30] (03CR) 10Eileen: [C: 03+2] "Yep - query speed is good" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857072 (https://phabricator.wikimedia.org/T323081) (owner: 10Ejegg) [22:23:33] right [22:25:14] other options: we have a log scraper that can extract specific numbers from well formatted logs [22:25:31] 10Fundraising-Analysis, 10Fundraising-Backlog, 10FR-Email: Diff Tool to compare two emails - Test vs Control - https://phabricator.wikimedia.org/T322891 (10DBu-WMF) Does but need to be within acoustic. My thought is when Acoustic generates a "view online" link we can compare two html pages [22:26:04] but that's going to run behind reality and be subject to prometheus's lack of time precision [22:28:27] (03PS2) 10Ejegg: Export donations qc stats every minute [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857074 (https://phabricator.wikimedia.org/T323162) [22:29:25] ejegg: we will need to increase the threshold now - note that your version includes ANY update - so will include thank yous etc [22:29:27] (03CR) 10Jgleeson: Export donations qc stats every minute (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857074 (https://phabricator.wikimedia.org/T323162) (owner: 10Ejegg) [22:29:52] eileen: oooh, right, that explains the much larger number [22:30:03] I guess I could have filtered by log_action='insert' too [22:30:04] although it's down to 426 in 5 mins now & set to 500 - whereas it was like 1700 before [22:30:25] so maybe 500 is still ok [22:30:41] I think it's ok to leave as 'any type' & see how it goes [22:30:50] after all any-type is activity [22:35:18] sure, makes sense [22:35:25] and better to be conservative for now [22:35:50] (03Merged) 10jenkins-bot: Query different table for queue backup [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857072 (https://phabricator.wikimedia.org/T323081) (owner: 10Ejegg) [22:39:27] eileen: did you deploy both of the cache fixes? [22:39:33] ejegg: yes [22:39:38] ah cool [22:39:40] no great improvement :-( [22:40:18] very choppy [22:40:59] (03CR) 10CI reject: [V: 04-1] Export donations qc stats every minute [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857074 (https://phabricator.wikimedia.org/T323162) (owner: 10Ejegg) [22:42:22] (03CR) 10Jgleeson: "One thing I'm worried about regarding this change is the risk of us losing data. Is it possible that we could reach 60s processing time, e" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857074 (https://phabricator.wikimedia.org/T323162) (owner: 10Ejegg) [22:44:34] (03CR) 10Ejegg: Export donations qc stats every minute (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857074 (https://phabricator.wikimedia.org/T323162) (owner: 10Ejegg) [22:45:01] ejegg: will you turn dedupe back on now? [22:45:11] sure thing eileen [22:45:25] oh, after deploying that backup check fix, i guess [22:45:51] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/857082 [22:46:20] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/857082 (owner: 10Ejegg) [22:49:21] ah right, deploy branch doesn't get automerged [22:51:05] (03PS1) 10Wfan: Filter out is_deleted = 1 for valid contact local options [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857083 (https://phabricator.wikimedia.org/T321251) [22:51:09] !log civicrm upgraded from d85589e8 to fa71f219 [22:51:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:52:03] !log re-enabled civicrm dedupe jobs [22:52:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:03:22] (03PS1) 10Eileen: Add name to wmffraud info file [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857086 [23:04:02] (03CR) 10Eileen: [C: 03+2] Filter out is_deleted = 1 for valid contact local options [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857083 (https://phabricator.wikimedia.org/T321251) (owner: 10Wfan) [23:04:02] heading out for a bit [23:09:07] (03CR) 10Cstone: CreateRecurringPaymentsProfile (031 comment) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/853318 (https://phabricator.wikimedia.org/T318881) (owner: 10Wfan) [23:11:19] 10Fundraising Sprint Undefined, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Review our queue speed - is it OK? - https://phabricator.wikimedia.org/T322791 (10jgleeson) I've been looking at the raw stats generated during tonight's traffic. There's a reasonably consistent negative correlation betwe... [23:15:56] eileen: it looks like the typical high processing rates we're used to might just be a function of low donation volume [23:16:30] there's a negative correlation between donations processed and processing rates [23:16:35] (03Merged) 10jenkins-bot: Filter out is_deleted = 1 for valid contact local options [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/857083 (https://phabricator.wikimedia.org/T321251) (owner: 10Wfan) [23:16:47] possibly due to reconnection overheads? [23:16:54] jgleeson: ah - yeah hard to parse that out - I 'feel like' we should be able to hit 7 somewhat consistently under load - but don't really have the data [23:17:18] & yeah - reconnection = warm everything up again [23:17:25] we're hitting 7 on short runs atm [23:17:34] dropping to 4 in full runs [23:17:51] full runs being close to the full 3min run time [23:17:55] yeah - so why slower on full runs - if it was reconnection overhead then short should be slower [23:18:39] blocking/waiting maybe [23:18:55] yeah - more interaction with other jobs? [23:19:14] it's skewed a little as different types of imports take longer than others notably recurrings [23:19:38] but if we can exclude those somehow and compare like for like and demonstrate the degrading we might be able to fix it [23:20:25] we might be able to identify patterns by gateway too [23:21:04] I'm gonna spend a bit of time tomorrow on gafana to see if we can do more with the stats we have [23:21:37] feels like the batch size per run and dontion counts by gateway per run would help [23:30:46] I'm seeing spates of deadlocks now - I suspect it gets slow when deadlocks start happening - which reflects 2 jobs at once [23:33:31] deadlocks are very much on contribution_xtra - not sure if that warrants some investigation - I think we use that for last orig currency update [23:38:57] hmm [23:39:17] feels like the tide is coming on for sure this year [23:39:24] coming in* [23:39:26] ok bed [23:39:28] bye for now all