[00:01:32] If anyone does want to tackle reviewing https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/772943 - that acoustic config is actually the trickiest part :-) [01:15:35] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops, 10User-greg: Voter party field - do we need to drop from backups or can it age out? - https://phabricator.wikimedia.org/T304360 (10greg) a:05Ejegg→03greg I'm asking on the relevant email thread. [03:07:10] 10Fundraising Sprint e^🥧👀=yum, 10Fundraising-Backlog, 10FR-Civi-Prospect: civi Time out - https://phabricator.wikimedia.org/T304530 (10Ejegg) @NNichols oops, I was looking at errors from a few days ago - looks like Civi might have been giving you problems when you tried to export some of these same users? An... [03:42:10] (03PS7) 10Ejegg: Fall back to invoice ID to find refund parent [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/674470 (https://phabricator.wikimedia.org/T277244) [03:44:14] (03CR) 10Ejegg: [C: 03+1] "Thanks for fixing this patch, cstone! This looks good and works on my machine. Want to give it the final +2 ?" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/674470 (https://phabricator.wikimedia.org/T277244) (owner: 10Ejegg) [03:45:27] whew cstone I finally got around to going through all the steps [03:45:30] worked like a charm! [04:01:33] 10Fundraising Sprint e^🥧👀=yum, 10Fundraising-Backlog, 10FR-Civi-Prospect: civi Time out - https://phabricator.wikimedia.org/T304530 (10Eileenmcnaughton) @NNichols I plugged through the spreadsheet trying to find an issue or pattern - but I found pretty consistently I could import around 2000 rows in a batch... [05:20:22] AndyRussG: I just pushed up credentials to the acoustic 'sandbox' account [05:24:53] eileen: ah cool! to config-private? [05:25:06] AndyRussG: yep [05:25:15] Just adding notes to our docs [05:52:03] eileen: cool .... almost sleepy time here, but I'll check it all out tomorrow, okok? [05:52:08] and thx! [05:52:34] AndyRussG: yeah sure - I am finishing up too [05:52:58] :) [06:32:11] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Amazon_endpoint_critical 2 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [06:37:11] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [11:11:46] (03CR) 10Damilare Adedoyin: [C: 03+2] Set maximum/minimum configurable value for sample rate of CentralNotice events [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/773187 (https://phabricator.wikimedia.org/T303326) (owner: 10Damilare Adedoyin) [11:14:12] (03Merged) 10jenkins-bot: Set maximum/minimum configurable value for sample rate of CentralNotice events [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/773187 (https://phabricator.wikimedia.org/T303326) (owner: 10Damilare Adedoyin) [13:32:15] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5036 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [13:37:15] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1527 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [13:42:15] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 686804 Threads: 11 Questions: 23378143 Slow queries: 339 Opens: 79009932 Flush tables: 1 Open tables: 200 Queries per second avg: 34.039 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [13:42:35] 10Fundraising Sprint e^🥧👀=yum, 10Fundraising-Backlog, 10FR-Civi-Prospect: civi Time out - https://phabricator.wikimedia.org/T304530 (10NNichols) @Eileenmcnaughton Thank you for getting that in for us! We will need to make these kind of large import updates on a regular basis in the future [15:05:38] 10Fundraising Sprint e^🥧👀=yum, 10Fundraising-Backlog: Phone number not populating in Civi from import files - https://phabricator.wikimedia.org/T303886 (10MDemosWMF) Oh wow! Glad we caught this. I double checked to make sure we hadn't changed the field name at some point, but looks like it has always been 'Pho... [15:34:48] 10Fundraising-Backlog, 10FR-Brazil, 10FR-dlocal: PIX workflow in Dlocal/Brazil - https://phabricator.wikimedia.org/T304014 (10DStrine) Circling back on this. We discussed this in the Tuesday meeting. This is going to be too complicated to work on at this time considering other priorities. The way dlocal offe... [15:53:19] (03CR) 10Cstone: [C: 03+2] "Just a year delayed" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/674470 (https://phabricator.wikimedia.org/T277244) (owner: 10Ejegg) [16:10:12] (03Merged) 10jenkins-bot: Fall back to invoice ID to find refund parent [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/674470 (https://phabricator.wikimedia.org/T277244) (owner: 10Ejegg) [16:15:06] 10Fundraising Sprint Anti-matter doesn't matter, 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising Sprint e^🥧👀=yum, 10Fundraising-Backlog, and 4 others: Enable South Africa through Dlocal - https://phabricator.wikimedia.org/T293508 (10DStrine) @EMartin and @HNordeenWMF we just have a few tiny thing... [16:50:56] 10Fundraising Sprint Anti-matter doesn't matter, 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising Sprint e^🥧👀=yum, 10Fundraising-Backlog, and 4 others: Enable South Africa through Dlocal - https://phabricator.wikimedia.org/T293508 (10HNordeenWMF) Thanks for the update @DStrine ! Sounds good about... [16:59:04] 10Fundraising-Backlog, 10FR-Brazil, 10FR-dlocal: PIX workflow in Dlocal/Brazil - https://phabricator.wikimedia.org/T304014 (10EMartin) Noted @dstrine. I will track for after the Dlocal re-integration. Thanks for looking at it. Evelyn Martin (She/Her) Sr. Manager Donation Processing | Wikimedia Foundatio... [17:00:19] 10Fundraising-Backlog, 10FR-dlocal: Turn on dlocal as default for South Africa - https://phabricator.wikimedia.org/T304627 (10DStrine) [17:01:37] 10Fundraising Sprint Anti-matter doesn't matter, 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising Sprint e^🥧👀=yum, 10Fundraising-Backlog, and 4 others: Enable South Africa through Dlocal - https://phabricator.wikimedia.org/T293508 (10DStrine) You can test earlier if you want. I have made the task... [17:08:34] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops, 10User-greg: Voter party field - do we need to drop from backups or can it age out? - https://phabricator.wikimedia.org/T304360 (10greg) Confirmed that we do **not** need to worry about backups at this time. [17:08:50] 10Fundraising Sprint Anti-matter doesn't matter, 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising Sprint e^🥧👀=yum, 10Fundraising-Backlog, and 4 others: Enable South Africa through Dlocal - https://phabricator.wikimedia.org/T293508 (10EMartin) David, UR and PE are currently default, correct? We al... [17:14:03] 10Fundraising-Backlog, 10FR-dlocal, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14): Make dlocal default for PE UY - https://phabricator.wikimedia.org/T303207 (10DStrine) @EMartin FYI default is ingenico right now until the 1 hour test and a conversation. [17:14:40] 10Fundraising Sprint Anti-matter doesn't matter, 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising Sprint e^🥧👀=yum, 10Fundraising-Backlog, and 4 others: Enable South Africa through Dlocal - https://phabricator.wikimedia.org/T293508 (10DStrine) PE and UY are ingenico at the moment. This is the task... [17:44:32] fr-tech I'm going to deploy civi to get that audit fix out [17:45:07] (03PS1) 10Cstone: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/773605 [17:47:33] (03Abandoned) 10Cstone: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/773605 (owner: 10Cstone) [17:49:32] (03PS1) 10Cstone: Switch back default audit location [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/773606 [17:50:00] fr-tech if someone could look at that one real quick ^ my testing settings accidentally got merged [17:50:39] just switching it back to what it was before [17:51:50] cstone: can u link the commit or change ID where they got changed accidentally on the commit message perhaps? [17:51:56] sure [17:52:44] (03PS2) 10Cstone: Switch back default audit location [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/773606 [17:53:40] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/773606/ do you want this to be plus 2? [17:54:41] (03CR) 10Wfan: [C: 03+2] "if you just want roll back then looks fine" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/773606 (owner: 10Cstone) [17:55:06] yeah the base audit path directory got committed even though i actively thought I had not commited it [17:55:11] I see you just want to roll it back. so I plus two for this :) [17:55:20] thanks! [17:55:29] 😃 np [17:56:01] thx cstone wfan :) [17:56:12] thanks AndyRussG wfan ! [17:56:18] :) are those settings something we should make standard in fr-dev? [17:56:29] they are actually controlled by the UI [17:56:33] I just didn't learn that till later haha [17:56:49] oh huh fun [17:57:11] and that patch elliott had for it I think deals with them [17:57:16] still something we should make default work properly under Docker? [17:57:44] AndyRussG: https://github.com/civicrm/civicrm-buildkit/pull/680 [17:59:19] i see how i broke it I meant to update the other file and the wrong one got in bah [17:59:32] broke it as in commited the settings [18:03:52] ahh oki thx cstone [18:09:12] (03Merged) 10jenkins-bot: Switch back default audit location [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/773606 (owner: 10Cstone) [18:19:59] haha cursed laptop activities now the mouse click is perma clicked [18:25:50] (03PS1) 10Cstone: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/773615 [18:28:57] (03CR) 10Cstone: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/773615 (owner: 10Cstone) [18:38:07] (03CR) 10Cstone: [C: 03+2] "recheck" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/773615 (owner: 10Cstone) [18:41:44] !log civicrm revision changed from b6ceb722 to 4e5b37c3 [18:41:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:42:42] 10Fundraising-Backlog, 10fr-donorservices: Subsequent recurring donation refunds not showing up in civi - https://phabricator.wikimedia.org/T277244 (10Cstone) [18:42:44] 10Fundraising Sprint Discworld reformatted as ntfs, 10Fundraising Sprint e^🥧👀=yum, 10Fundraising-Backlog: Transaction canceled in Ingenico but not in Civi - https://phabricator.wikimedia.org/T301480 (10Cstone) [18:46:38] 10Fundraising Sprint e^🥧👀=yum, 10Fundraising-Backlog, 10FR-Civi-Prospect: civi Time out - https://phabricator.wikimedia.org/T304530 (10DStrine) Yeah can anyone explain what happened here? Civi users are used to importing much larger files on a regular basis. We need to support that use case. [21:25:37] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Ingenico, 10FR-Sweden, 10Recurring-Donations: Sweden MonthlyConvert_Thank_You Email - https://phabricator.wikimedia.org/T301284 (10DStrine) 05Open→03Resolved [21:58:46] 10fundraising-tech-ops: install/configure frauth1002 - https://phabricator.wikimedia.org/T299068 (10Dwisehaupt) [21:59:23] 10fundraising-tech-ops: install/configure frauth1002 - https://phabricator.wikimedia.org/T299068 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt host built and in place. Any further issues should be in new tasks. [21:59:51] 10Fundraising Sprint e^🥧👀=yum, 10Fundraising-Backlog, 10FR-Civi-Prospect: civi Time out - https://phabricator.wikimedia.org/T304530 (10Eileenmcnaughton) @DStrine most of our imports are done through our custom code - this is the main civi ui - for the custom code we can toggle the batch right down in the UI-... [21:59:57] 10fundraising-tech-ops, 10Patch-For-Review: install/configure civi1002 - https://phabricator.wikimedia.org/T296409 (10Dwisehaupt) [22:00:12] 10fundraising-tech-ops, 10Patch-For-Review: install/configure civi1002 - https://phabricator.wikimedia.org/T296409 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt host built and in place. Any further issues should be in new tasks. [22:00:39] 10fundraising-tech-ops: install/configure frpm1002 - https://phabricator.wikimedia.org/T299069 (10Dwisehaupt) [22:01:00] 10fundraising-tech-ops: install/configure frpm1002 - https://phabricator.wikimedia.org/T299069 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt host built and in place. Any further issues should be in new tasks. [22:11:47] 10Fundraising Sprint e^🥧👀=yum, 10Fundraising-Backlog, 10FR-Civi-Prospect: civi Time out - https://phabricator.wikimedia.org/T304530 (10Dwisehaupt) @Eileenmcnaughton Just for clarity, the php config setting that is at 30 seconds is max_execution_time (https://www.php.net/manual/en/info.configuration.php#ini.m... [22:12:39] 10Fundraising Sprint e^🥧👀=yum, 10Fundraising-Backlog, 10FR-Civi-Prospect: civi Time out - https://phabricator.wikimedia.org/T304530 (10Eileenmcnaughton) @Dwisehaupt yes! [22:20:55] eileen: you around? [22:21:02] dwisehaupt: yep [22:22:00] ok. the max_execution_time is set to 60 currently for php::mod::apache2, but only the default 30 for php::mod::cgi. [22:22:41] i'll bump it to 60 for the cgi mod and it will be consistent. [22:23:20] hmm so is mod cgi apache web ui & mod cgi command line? [22:23:34] 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: MediaWiki:Centralnotice: contradiction for upper bound: Centralnotice-banner-history-logger-rate-help vs. Centralnotice-impression-events-sample-rate-field - https://phabricator.wikimedia.org/T304638 (10AndyRussG) [22:23:43] well, there are 3, for extra confusion. [22:24:07] there is mod apache2, mod cgi, and mod cli [22:24:10] I can actually try re-importing that file (even though it is imported it can still run again - it just won't change much ) - so I can test [22:25:09] so the first two apply to the web/cgi side, but different parts of it. the cli portion will stay at 30 seconds for now. if we want to change that i'll have to template out that portion of the file. [22:25:57] ok cool [22:26:07] if you have changed it I will retry the import [22:29:07] ok. just changed and rolled the puppet run. have a go again. [22:30:12] 10Fundraising Sprint e^🥧👀=yum, 10Fundraising-Backlog, 10FR-Civi-Prospect: civi Time out - https://phabricator.wikimedia.org/T304530 (10Dwisehaupt) Updated the timeout for php::mod::cgi to 60 seconds. ` [frack::puppet] 4fc65d09 update php::mod::cgi max_execution_time to 60 for civicrm ` [22:35:40] OK - its running - started 11.35 [22:36:00] it says it will take 94 minutes! [22:36:37] um. wow. [22:38:24] 550 rows 2% completed 11.38 [22:41:15] 1150 rows 5$ completed 11.41 [22:45:02] 1950 9% completed 11.45 [22:45:34] dwisehaupt: dang it just timed out [22:46:42] hmmm... [22:48:38] I can't see anything in logs to explain [22:49:17] the time out is 504 Gateway Time-out [22:51:15] yeah. that's coming from the nginx side. [22:51:37] I thought nginx might say 'something' about it tho [22:52:17] note that every 50 rows it does a new ajax call [22:54:02] not much extra, but there is a proxy timeout of 600s. [22:54:22] which is 10mins which is pretty much spot on. [22:56:02] dwisehaupt: ah - so why do we hit that? [22:56:37] well, nginx is the front end layer. everything passes through there back to the apache/php layer. [22:56:57] it controls certain things like redirects and ssl client certs at the front end. [22:57:26] so that timeout would come into place since the connection is flowing through there. [23:00:25] is there some other way to run this large process. since guaranteeing a consistent connection to a browser for 95 minutes seems fraught with peril. [23:00:38] hmm - I thought the fact that multiple ajax requests are in play would reset that? [23:01:15] re: ajax, yeah. i would think so too. i don't know enough about that to say for sure. [23:01:59] it's a shame it doesn't log something when it times out -then we would know [23:04:57] yeah. the logs are odd. [23:05:40] oh. i think i see what's happening, maybe. [23:06:17] so, i can see every 5 seconds during the process that there is a GET https://civicrm.wikimedia.org/civicrm/ajax/status?id=32e9edb0601fb1493d4bfe791363da29 [23:06:33] but those all show up as separate entries. [23:08:00] that seems to just check a file that tracks status (ug) [23:08:14] the post initally shows up as a 302, and then gets the 504 after the timeout. [23:10:20] oh. actually this is odd. the 504 doesn't hit the logs until 23:56:14 [23:12:30] hmmm... yeah. these logs may be confusing/misleading me the way i'm looking at them. [23:12:54] i'm going to pull a chunk aside and see if i can piece it together. [23:14:25] dwisehaupt: I think the process might continue in the back ground even after the 504 - is it worth upping the max_execution time temporarily higher to rule it out - or moving the proxy to 15 mins temporarily to rule that out (or both) [23:15:42] there are 5 settings mentioned here - https://supporthost.com/504-gateway-timeout/ [23:15:43] - but basically they fall into max execution time & proxy time out [23:16:50] there are a lot of drupal error entries in the log. [23:17:09] Undefined index: is_required in CRM_Contact_Import_Parser_Contact->deprecated_validate_formatted_contact() (line 2375 of /srv/org.wikimedia.civicrm/civicrm/CRM/Contact/Import/Parser/Contact.php). [23:18:37] dwisehaupt: yeah I was looking at those - annoying but prob not the cause? [23:19:34] probably not the cause, but definitely makes parsing the logs more difficult. [23:27:54] heh. i exit out of the log file, look at something for a second and then come back in and got lost. since daylight confusion time has my brain off of where utc should be. [23:29:50] :-) [23:30:05] dwisehaupt: maybe log a phab on that notice - would be good to fix regardless [23:31:56] sure thing. [23:37:16] 10Fundraising-Backlog: Clean up drupal errors related to the contact import parser - https://phabricator.wikimedia.org/T304663 (10Dwisehaupt)