[01:34:20] (03PS1) 10Eileen: Towards moving RefundTest to RefundQueueConumerTest [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036330 (https://phabricator.wikimedia.org/T365415) [01:44:45] (03CR) 10CI reject: [V:04-1] Towards moving RefundTest to RefundQueueConumerTest [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036330 (https://phabricator.wikimedia.org/T365415) (owner: 10Eileen) [01:48:07] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fail Mail (civi1002) run-job: Get recipient data from Silverpop failed with code 1 - https://phabricator.wikimedia.org/T365544#9836346 (10Eileenmcnaughton) I had a go at logging into Acoustic ... and failed - I'm assuming the issue here is the credentials... [01:50:51] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Recurring Donor (segment_id 400) records have conflicting status Deep Lapsed (status_id 60) - https://phabricator.wikimedia.org/T365639#9836347 (10Eileenmcnaughton) I guess we will be pushing up the right statuses now but not re-actively updating them all... [01:57:03] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: New Segmentation Framework not quite usable yet - https://phabricator.wikimedia.org/T363527#9836352 (10Eileenmcnaughton) Actually this is not quite the same issue - we calculate the Donor Statuses based on the donations they have in CiviCRM. It's a pretty... [02:09:05] (03PS1) 10Eileen: Superficial clean up in tests, exception naming [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036332 (https://phabricator.wikimedia.org/T363965) [02:09:07] (03PS1) 10Eileen: Superficial clean up - we have standardised on the entity being CamelCase in the ids array [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036333 (https://phabricator.wikimedia.org/T363965) [02:11:31] (03PS1) 10Novem Linguae: add namespace to PHP use statement [extensions/FundraisingTranslateWorkflow] - 10https://gerrit.wikimedia.org/r/1036334 [02:15:36] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-Translate, 10ci-test-error (WMF-deployed Build Failure): phpunit test failure for extension FundraisingTranslateWorkflow - https://phabricator.wikimedia.org/T357804#9836360 (10Novem_Linguae) Tests pass in localhost with `docker... [02:20:54] (03CR) 10CI reject: [V:04-1] add namespace to PHP use statement [extensions/FundraisingTranslateWorkflow] - 10https://gerrit.wikimedia.org/r/1036334 (owner: 10Novem Linguae) [02:27:40] (03PS2) 10Eileen: Superficial clean up - we have standardised on the entity being CamelCase in the ids array [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036333 (https://phabricator.wikimedia.org/T363965) [02:27:40] (03PS1) 10Eileen: Minor cleanup, join if clauses together for clarity [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036336 [02:29:08] (03CR) 10CI reject: [V:04-1] Superficial clean up - we have standardised on the entity being CamelCase in the ids array [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036333 (https://phabricator.wikimedia.org/T363965) (owner: 10Eileen) [02:29:40] (03CR) 10CI reject: [V:04-1] Superficial clean up in tests, exception naming [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036332 (https://phabricator.wikimedia.org/T363965) (owner: 10Eileen) [02:46:32] (03PS2) 10Eileen: Superficial clean up in tests, exception naming [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036332 (https://phabricator.wikimedia.org/T363965) [02:46:32] (03PS2) 10Eileen: Minor cleanup, join if clauses together for clarity [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036336 [02:46:32] (03PS3) 10Eileen: Superficial clean up - we have standardised on the entity being CamelCase in the ids array [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036333 (https://phabricator.wikimedia.org/T363965) [02:46:33] (03PS1) 10Eileen: Add test & handling for reversed names [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036338 [02:50:24] (03CR) 10CI reject: [V:04-1] Superficial clean up - we have standardised on the entity being CamelCase in the ids array [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036333 (https://phabricator.wikimedia.org/T363965) (owner: 10Eileen) [03:07:46] (03CR) 10CI reject: [V:04-1] Add test & handling for reversed names [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036338 (owner: 10Eileen) [03:08:44] (03CR) 10CI reject: [V:04-1] Superficial clean up - we have standardised on the entity being CamelCase in the ids array [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036333 (https://phabricator.wikimedia.org/T363965) (owner: 10Eileen) [03:10:12] (03CR) 10CI reject: [V:04-1] Minor cleanup, join if clauses together for clarity [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036336 (owner: 10Eileen) [03:13:52] (03CR) 10CI reject: [V:04-1] Superficial clean up in tests, exception naming [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036332 (https://phabricator.wikimedia.org/T363965) (owner: 10Eileen) [03:33:45] (03PS3) 10Eileen: Minor cleanup, join if clauses together for clarity [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036336 [03:33:45] (03PS3) 10Eileen: Superficial clean up in tests, exception naming [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036332 (https://phabricator.wikimedia.org/T363965) [03:33:45] (03PS4) 10Eileen: Superficial clean up - we have standardised on the entity being CamelCase in the ids array [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036333 (https://phabricator.wikimedia.org/T363965) [03:33:46] (03PS2) 10Eileen: Add test & handling for reversed names [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036338 [03:37:46] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_39) - 10https://gerrit.wikimedia.org/r/1035975 (owner: 10L10n-bot) [03:54:34] (03CR) 10CI reject: [V:04-1] Superficial clean up - we have standardised on the entity being CamelCase in the ids array [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036333 (https://phabricator.wikimedia.org/T363965) (owner: 10Eileen) [03:56:25] (03CR) 10CI reject: [V:04-1] Superficial clean up in tests, exception naming [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036332 (https://phabricator.wikimedia.org/T363965) (owner: 10Eileen) [03:56:31] (03CR) 10CI reject: [V:04-1] Add test & handling for reversed names [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036338 (owner: 10Eileen) [04:00:37] (03PS1) 10Eileen: APiv4 test conversion (baby step) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036341 [04:02:29] (03CR) 10CI reject: [V:04-1] Minor cleanup, join if clauses together for clarity [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036336 (owner: 10Eileen) [04:22:53] (03CR) 10CI reject: [V:04-1] APiv4 test conversion (baby step) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036341 (owner: 10Eileen) [04:59:15] (03PS4) 10Eileen: Superficial clean up in tests, exception naming [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036332 (https://phabricator.wikimedia.org/T363965) [04:59:15] (03PS5) 10Eileen: Superficial clean up - we have standardised on the entity being CamelCase in the ids array [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036333 (https://phabricator.wikimedia.org/T363965) [04:59:15] (03PS4) 10Eileen: Minor cleanup, join if clauses together for clarity [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036336 [04:59:16] (03PS3) 10Eileen: Add test & handling for reversed names [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036338 [04:59:17] (03PS2) 10Eileen: APiv4 test conversion (baby step) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036341 [05:21:15] (03CR) 10CI reject: [V:04-1] Add test & handling for reversed names [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036338 (owner: 10Eileen) [05:21:53] (03CR) 10CI reject: [V:04-1] Minor cleanup, join if clauses together for clarity [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036336 (owner: 10Eileen) [05:21:55] (03CR) 10CI reject: [V:04-1] APiv4 test conversion (baby step) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036341 (owner: 10Eileen) [06:25:12] (03PS3) 10Eileen: APiv4 test conversion (baby step) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036341 [06:31:58] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_40) - 10https://gerrit.wikimedia.org/r/1036398 (owner: 10L10n-bot) [06:37:20] (03PS1) 10Eileen: Pull code from upstream - this is in upstream git repo [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036481 [07:37:05] (03PS2) 10Eileen: Pull code from upstream - this is in upstream git repo [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036481 [07:37:05] (03PS5) 10Eileen: Superficial clean up in tests, exception naming [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036332 (https://phabricator.wikimedia.org/T363965) [07:37:06] (03PS6) 10Eileen: Superficial clean up - we have standardised on the entity being CamelCase in the ids array [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036333 (https://phabricator.wikimedia.org/T363965) [07:37:06] (03PS4) 10Eileen: APiv4 test conversion (baby step) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036341 [07:37:07] (03PS1) 10Eileen: Add test & fix for reversed names [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036552 [07:37:37] (03Abandoned) 10Eileen: Minor cleanup, join if clauses together for clarity [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036336 (owner: 10Eileen) [07:37:44] (03Abandoned) 10Eileen: Add test & handling for reversed names [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036338 (owner: 10Eileen) [07:57:09] (03PS2) 10Eileen: Towards moving RefundTest to RefundQueueConumerTest [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036330 (https://phabricator.wikimedia.org/T365415) [09:19:08] (03PS1) 10Eileen: Add test & fix for issue with type hints [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036576 (https://phabricator.wikimedia.org/T363965) [09:20:07] (03PS1) 10Eileen: Superficial clean up [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036577 [11:11:27] woah failmail storm [11:11:50] thank you emails [11:11:53] backing up maybe [11:15:15] looks like the thank you emails started backing up yesterday about 9am UTC https://frmon.wikimedia.org/d/e0a72c1b-cd9e-4a25-b781-c981d51f58db/jack-big-english-dash?orgId=1&refresh=1m&from=now-2d&to=now&viewPanel=21 [11:40:25] 10Wikimedia-Fundraising-Banners, 10donate.wikimedia.org: Add Carte Bancaire logo and enable card type logos in France - https://phabricator.wikimedia.org/T204545#9837363 (10Pcoombe) 05Open→03Declined Closing, my understanding is that Carte Bancaire use would solely be on the backend and the logo doesn'... [11:42:25] hmm the thank you emails are going out 30 every 3 minutes or so but it's not making much of a dent in the backlog [11:44:46] 06Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface: Use numeric keyboard on mobile for credit card number and security code - https://phabricator.wikimedia.org/T169207#9837368 (10Pcoombe) 05Open→03Resolved Confirmed that Adyen and dLocal both do this now 👍 [11:48:39] 06Fundraising-Backlog, 10FR-Adyen, 10MediaWiki-extensions-DonationInterface: Make Adyen default gateway for Ukraine - https://phabricator.wikimedia.org/T140086#9837380 (10Pcoombe) 05Open→03Resolved Adyen is default in most countries now, plus we aren't fundraising in Ukraine [11:54:25] ok there's about 9000 donations waiting to get their thank you emails [11:54:33] that's not the end of the world [12:02:46] I wonder if we could start sending more emails through frmx2001 as that seems to be processing faster based on the log output. currently we're only sending via that exchange once an hour [12:06:20] fr-tech I guess we can't run both thank you send jobs simultaneously as it would risk them both picking up the same job... I'm not sure if we have a locked/pending flag [12:19:59] looks like the recurring traffic has finished for now and the thank you queue is dropping [12:20:11] Hi jgleeson ! Dallas was looking into this a little yesterday - can you see the backscroll on fredgy? [12:21:26] thanks anilk [12:24:00] if we introduce a 'processing_status' style flag to the thank you email jobs then we could sending mail through the two exchanges simultaneously doubling our processing rate, all things staying the same [12:24:24] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Email: Civi to Acoustic Export: Job "MASTER SUPPRESSION LIST" failing every other day. - https://phabricator.wikimedia.org/T366059 (10ppenloglou) 03NEW [13:13:14] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14411 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [13:18:12] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14711 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [13:23:16] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15011 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [13:28:14] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15311 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [13:33:18] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15611 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [13:37:40] hi fr-tech! [13:37:45] howdy ejegg [13:37:53] looks like my IRC box rebooted so I have no backscroll [13:37:59] i feel so blind [13:38:05] how's things going? [13:38:14] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15911 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [13:38:17] I see there were a lot of TY mailer failmails [13:38:32] we've got a thank you email backup which is processing slowly [13:38:43] thread here https://wikimedia.slack.com/archives/C045WH0QYS2/p1716873408956839 [13:38:48] thanks! [13:39:29] ejegg: my instinct is telling me something is off but i can't see what. a quick fix might be sending for mail to frmx2001 as that appears to be processing faster [13:39:34] wow, only 660 per hour? [13:39:36] s/for/more/ [13:39:39] yeah [13:39:47] there is a 30 per run limit being set [13:39:48] yikes [13:39:53] wait, really? [13:40:03] I think so [13:40:15] ok, we should definitely increase that [13:40:22] ACKNOWLEDGEMENT - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15911 Jeff_Green database dump still running https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [13:41:31] jgleeson: ok, maybe it's just the batch limit [13:41:43] err wait no [13:41:54] i was looking at minutes and thinking they were seconds [13:42:14] so it really is taking the full 3 minute run to send 30 or so mails [13:42:34] Jeff_Green: any idea what would slow outgoing mail so much on frmx1001? [13:42:55] apparently although it seems like a pretty consistent low rate across all jobs [13:43:17] so let's add some logging around the query to see if that's slow [13:43:27] do we have to be in the drupal dir to see those drupal vars / civicrm_settings [13:43:58] jgleeson@civi1002:/var/log/process-control/thank_you_mail_send_smtp_frmx2001$ drush @wmff vget | grep thank_you_batch [13:44:00] thank_you_batch: '1450' [13:44:02] thank_you_batch_time: '173' [13:44:04] hmm [13:44:08] maybe there isn't a limit [13:44:18] jgleeson: ah those drush vars are obsolete now [13:44:35] let's see, what's the analogous cv command to get the civi setting... [13:44:38] https://github.com/wikimedia/wikimedia-fundraising-crm/blob/824d317f4a28b26abca0bf568cce766b57cfe626/drupal/sites/all/modules/thank_you/thank_you.module#L304 [13:44:56] is that in the settings tbl? [13:45:14] its in civicrm.civicrm_setting I think [13:46:33] name: thank_you_batch [13:46:35] value: s:4:"1450"; [13:46:39] looks like they are the same [13:47:11] MariaDB [civicrm]> SELECT * FROM civicrm.civicrm_setting where name LIKE 'thank_you%' \G [13:47:28] oh wait [13:47:30] name: thank_you_batch_time [13:47:32] value: s:3:"173"; [13:47:39] yeo [13:47:40] yep [13:47:44] cv api3 setting.get [13:48:02] I guess that's about right, 3m with change [13:48:03] for the cli [13:48:09] ok, so it's not a batch limit [13:48:19] and in the logs we do see it taking the full time [13:48:26] let's log around the db query [13:48:48] dallas mentioned yesterday on fredge that he was seeing some overhead in the connection from civi to the mail exchange server [13:49:21] I caught up with the back scroll from yesterday after a nudge from anilk [13:51:21] i saw the failmail yesterday but i had a bunch of friends over that I hadn't seen in months [13:51:30] ... [13:52:19] huh, there's some deploy backlog [13:52:31] I kindof don't want to send that out right now [13:52:45] I'm going to merge this debug timing right onto the deploy branch [13:52:56] then we can revert it when we're ready to deploy the real code [13:52:58] fun [13:53:13] (03PS1) 10Ejegg: Logging around email sends [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036675 [13:53:18] want to give it a quick look? ^^^ [13:53:46] there's some other stuff that happens, like the civimail records [13:53:52] which I didn't put logging around [13:54:04] (03CR) 10Jgleeson: [C:03+2] "LGTM" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036675 (owner: 10Ejegg) [13:54:09] thanks! [13:54:13] (03CR) 10Ejegg: [V:03+2] Logging around email sends [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036675 (owner: 10Ejegg) [13:54:47] so that batch limit of 173 is gonna keep causing failmail I think [13:54:54] how so? [13:55:02] the time limit? [13:55:10] actually it won't [13:55:13] I forgot how to count [13:55:17] 3m=180 :) [13:55:27] my brain picked 160 at firsdt [13:55:29] -d [13:55:59] maybe 7 seconds isn't enough to clean up? [13:56:01] seems weird [13:56:18] ah, right, if it's hanging like 5 sec sending an email [13:56:22] it might go over the limit [13:56:50] and i think we had a little hack to account for startup time in the old drush scripts which we don't have in the new cv entrypoints [13:57:12] so there's another second or two or delay at the beginning that's not counted towards those 173 [13:57:44] !log fundraising civicrm upgraded from 6c1fdd4f to e2dc8f4e [13:57:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:59:27] Today's seed for your music mix: Wanda Jackson's version of "Rum and Coca Cola" [14:00:41] hmm in the logs I'm not seeing huge delays between events [14:00:43] "...Working for the Yankee dollar" [14:01:53] hmm, it's almost a second between 'finished query' and 'sending ty email' [14:01:57] so that's the time to render [14:02:02] I guess I'm looking at the first [14:02:08] should hopefully be faster later [14:02:31] anyway, the query seems like 2ms tops [14:03:02] so that rules out issues with the contact & contribution table query [14:03:14] RECOVERY - check_mysql on frdb1004 is OK: Uptime: 1020864 Threads: 5 Questions: 38710686 Slow queries: 475 Opens: 4476 Open tables: 1155 Queries per second avg: 37.919 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [14:03:30] I've been monitoring the latest logs across the different jobs by running this on civi1002 [14:03:35] `ls -tr /var/log/process-control/thank_you_mail_send_smtp_frmx1001/* | tail -n1 | xargs tail -f -n500 | bat -ppl ps1` [14:04:06] wow, there's a huge gap between the end of the query at 2024-05-28 13:58:25,230 and sending the email at 2024-05-28 13:58:34,739 [14:04:14] ok, so the rendering may be really slow? [14:04:15] looking [14:04:24] let's add more logging around the rendering calls [14:04:57] maybe something with the token system changed in the last 4 point versions of Civi [14:05:05] oh I see it here also [14:05:06] 2024-05-28 13:58:14,906 ERROR civicrm.wmf.INFO: Starting query for contribution 106435097 [14:05:08] 2024-05-28 13:58:14,907 ERROR civicrm.wmf.INFO: Finished query for contribution 106435097 [14:05:10] 2024-05-28 13:58:24,030 ERROR civicrm.wmf.INFO: thank_you: Sending ty email to: [14:05:15] 10 second gap [14:05:19] yep yep [14:05:36] I'll roll back the current timing patch and make a new one [14:05:57] oh interesting, I didn't see it at the top but it's slowing down as the job runs [14:06:14] (03PS1) 10Ejegg: Revert "Logging around email sends" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036677 [14:06:16] (03CR) 10Ejegg: [C:03+2] Revert "Logging around email sends" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036677 (owner: 10Ejegg) [14:06:31] the lag grows from 1-2 3-4 to 10 [14:06:35] seconds [14:07:02] (03Merged) 10jenkins-bot: Revert "Logging around email sends" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036677 (owner: 10Ejegg) [14:07:33] the 2001 and 60 jobs will have run recently also [14:07:36] we can check them [14:07:43] frmx2001 [14:08:17] yep same delay in frmx2001 job [14:08:18] 2024-05-28 14:01:50,399 ERROR civicrm.wmf.INFO: Starting query for contribution 106435321 [14:08:20] 2024-05-28 14:01:50,400 ERROR civicrm.wmf.INFO: Finished query for contribution 106435321 [14:08:20] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog: Paypal Giving Fund donations added to Civi under incorrect CID - https://phabricator.wikimedia.org/T363768#9838007 (10MDemosWMF) Noting that May's deposit is also showing on the other CID 63719152 [14:08:22] 2024-05-28 14:02:00,014 ERROR civicrm.wmf.INFO: thank_you: Sending ty email to: [14:08:24] so it's not the mail exchanges [14:12:24] (03PS1) 10Ejegg: More logging around message rendering [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036678 [14:12:41] right, jgleeson, logs seem to rule that out too [14:13:27] slowdowns in the Civi core message rendering and tokenization system are unfortunately probably going to be tougher to solve... [14:13:39] want to take a look at that other logging patch? [14:15:31] looking [14:17:16] ejegg: do we need a corresponding end for the render logger? [14:17:32] or do we infer that from something currently [14:17:39] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1036678/1/drupal/sites/default/civicrm/extensions/wmf-thankyou/Civi/Api4/Action/ThankYou/Render.php [14:17:51] sure, I could add that [14:18:23] (03PS2) 10Ejegg: More logging around message rendering [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036678 [14:20:44] ejegg: here's something to call out. the delay represented on this graph has almost halved over the last 3 hours or so meanwhile the queue of thank you emails to be sent has increased https://frmon.wikimedia.org/d/e0a72c1b-cd9e-4a25-b781-c981d51f58db/jack-big-english-dash?orgId=1&refresh=1m&from=now-3h&to=now&viewPanel=21 [14:21:17] (03CR) 10Jgleeson: [C:03+2] "LGTM" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036678 (owner: 10Ejegg) [14:21:37] we were getting recurring donations in earlier on around the time the delay peaked at 14h [14:21:59] (03Merged) 10jenkins-bot: More logging around message rendering [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036678 (owner: 10Ejegg) [14:22:13] 06Fundraising-Backlog, 10FR-Tech-Analytics, 10FR-tech-data-integrity: Non Campaign Emails in Database - https://phabricator.wikimedia.org/T361621#9838054 (10Ejegg) [14:22:43] hmm, could the rendering be suddenly doing more and more db lookups? [14:23:47] though if it's getting slower and slower over the course of the run that feels more like some coding error, where we're looping over longer and longer arrays with each repeat or the like [14:24:46] !log fundraising civicrm upgraded from e2dc8f4e to 7e998894 [14:24:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:24:59] so... we are on an RC version of Civi core [14:25:19] we could check to see what else has been merge to the release branch since we deployed [14:25:25] in case there's already a fix upstream [14:25:55] https://github.com/civicrm/civicrm-core/tree/5.74 [14:26:45] https://github.com/civicrm/civicrm-core/commits/5.74/ [14:27:41] ahh i think we also switched to smarty 5, right? [14:27:53] 2024-05-28 14:27:32,541 ERROR civicrm.wmf.INFO: Calling CRM_Core_TokenSmarty::render [14:27:54] so we could try going back to smarty 3 [14:27:55] 2024-05-28 14:27:35,811 ERROR civicrm.wmf.INFO: Finished CRM_Core_TokenSmarty::render [14:28:24] that's a bit of a delay alright [14:28:34] so let's see what smarty version we're using [14:29:08] 2024-05-28 14:28:40,848 ERROR civicrm.wmf.INFO: Calling CRM_Utils_Hook::alterMailContent [14:29:10] 2024-05-28 14:28:40,848 ERROR civicrm.wmf.INFO: Calling CRM_Core_TokenSmarty::render [14:29:12] 2024-05-28 14:28:45,274 ERROR civicrm.wmf.INFO: Finished CRM_Core_TokenSmarty::render [14:29:14] creeping up [14:29:27] oh huh, we are still on Smarty3 in production [14:29:32] so that didn't change [14:29:47] right, this poor man [14:29:51] 14Fundraising Sprint: ick(), 06Fundraising-Backlog, 10MW-1.43-notes (1.43.0-wmf.7; 2024-05-28): Enable 90 day snooze: Comms Preferences Centre update - https://phabricator.wikimedia.org/T358878#9838065 (10AKanji-WMF) @Ejegg: Noting that Mariana is still experiencing the line break issue above; I'm not ex... [14:29:52] s profiler [14:30:04] is helping narrow it down [14:30:07] deploy by deploy [14:30:46] can I play with CRM_Core_TokenSmarty locally by sending myself a ty email? [14:31:07] 14Fundraising Sprint: ick(), 06Fundraising-Backlog, 10MW-1.43-notes (1.43.0-wmf.7; 2024-05-28): Enable 90 day snooze: Comms Preferences Centre update - https://phabricator.wikimedia.org/T358878#9838068 (10Ejegg) @AKanji-WMF we've merged a patch for that linebreak issue but we haven't deployed it yet. I'l... [14:31:19] yep, that should be totally replicable locally jgleeson [14:32:06] (03PS1) 10Ejegg: Revert "More logging around message rendering" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036684 [14:32:08] (03CR) 10Ejegg: [C:03+2] Revert "More logging around message rendering" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036684 (owner: 10Ejegg) [14:32:55] +-----------------------------+ [14:32:57] | thank_you_email_queue_count | [14:32:59] +-----------------------------+ [14:33:01] | 9812 | [14:33:03] +-----------------------------+ [14:33:05] this is steadily rising [14:33:09] (03Merged) 10jenkins-bot: Revert "More logging around message rendering" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036684 (owner: 10Ejegg) [14:33:41] https://phabricator.wikimedia.org/P63445 [14:34:13] yep, guess it'll keep rising till we figure this out :( [14:34:25] I'm thinking about what to tell Sam [14:34:32] so... I guess one ugly fix would be to do shorter runs for now [14:34:35] 1 min each [14:34:58] let's put that bandaid on it and see if we get any improvement while we debug [14:35:01] to avoid is slowing down? [14:35:03] it* [14:35:18] yeah, it seems to be faster for the first minute, right? [14:35:23] it's odd that it's degrading during the run [14:35:25] yeah it is [14:35:41] maybe some singleton style cache [14:35:47] yarp [14:36:45] oh we still are using the drush entry point for the TY sender [14:36:59] i thought we had switched that over to cv as well but no [14:38:47] hmm, I guess i need to stop it temporarily [14:39:03] so we don't get overlapping runs when going from every 3 min to every 1 min [14:40:56] ACKNOWLEDGEMENT - check_missing_thank_yous on frdb1005 is CRITICAL: CRITICAL missing_thank_yous=7406 [critical =2000] Jeff_Green Investigation in process https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [14:42:19] ejegg: wanna do the p-c and I can update the SQL setting? [14:42:33] batch_time to 55 maybe? [14:42:46] ah, i think i was able to override it in the yaml [14:42:52] with --time-limit=55 [14:42:55] ah nice [14:43:31] at least, i HOPE we made the CLI option take priority :) [14:43:43] k, should be re-enabled [14:44:11] 15 in 1 min [14:44:18] 50% improvement? [14:44:37] ehh, it's something [14:44:40] it did ejegg [14:44:48] 14Fundraising Sprint: ick(), 06Fundraising-Backlog, 10MW-1.43-notes (1.43.0-wmf.7; 2024-05-28): Enable 90 day snooze: Comms Preferences Centre update - https://phabricator.wikimedia.org/T358878#9838106 (10AKanji-WMF) Ah apologies, @Ejegg, thank you. I'm working on the Paytm email which has a link to this... [14:44:52] 2024-05-28 14:43:56,401 ERROR civicrm.wmf.INFO: thank_you: Batch time limit (55 s) elapsed {"time_limit":"55"} [14:44:54] 2024-05-28 14:43:56,405 ERROR civicrm.wmf.INFO: thank_you: Sent 15 thank you emails. {"total_thank_you_emails":15} [14:44:56] still 2.4 sec delay on CRM_Core_TokenSmarty::render by the end of the run [14:45:09] beats 10 :P [14:45:13] so I guess I'll dig into that and add more logging [14:45:44] yeah I'm gonna debug it with a local ty send and see if there's any obvious areas [14:45:51] great! [14:45:54] I did see a preg_match but it looked harmless [14:47:09] 16 in that last run [14:47:53] oh that's weird ejegg [14:47:59] it seems to only effect some [14:48:10] 2024-05-28 14:46:48,732 ERROR civicrm.wmf.INFO: Calling CRM_Core_TokenSmarty::render [14:48:12] 2024-05-28 14:46:51,121 ERROR civicrm.wmf.INFO: Finished CRM_Core_TokenSmarty::render [14:48:14] but then the one after [14:48:23] 2024-05-28 14:46:54,361 ERROR civicrm.wmf.INFO: Calling CRM_Core_TokenSmarty::render [14:48:24] 2024-05-28 14:46:54,703 ERROR civicrm.wmf.INFO: Finished CRM_Core_TokenSmarty::render [14:48:35] I wonder if we have specific offending characters or replacements [14:48:56] i guess we could look at the specific thank you emails [14:49:11] brb coffee [14:52:40] huh [14:53:58] (03PS1) 10Ejegg: Log timings in TokenSmarty [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036685 [14:58:07] hmm, i want to log all the listeners for the 'civi.token.eval' event [14:58:19] but it looks like i'd have to add that logging inside the vendor dir [14:59:09] actually, let me set a breakpoint locally to see if there's anywhere in Civi that the symfony code calls before it goes right to the listener fns [15:00:32] there's a Subscriber thing [15:00:37] let's see how that works [15:01:07] ejegg: the queue delay dropped 30 minutes in the last 15 vs the previous 30 min drop taking 1h [15:01:16] good signs [15:01:40] we also got 17 out in the last 1 min run so it is pushing harder [15:02:45] well, that's a good start! [15:03:30] hmm, ok, eventSubscriber is basically a different flavor of listener, that the dispatcher just calls once on each dispatch of a single event [15:03:59] i want something that is aware of all the other listeners and knows how long each one takes to respond to the event [15:04:43] oho, ,there is a Traceable Event Dispatcher [15:04:59] https://symfony.com/doc/current/components/event_dispatcher/traceable_dispatcher.html [15:05:07] but... can we make civi use one of those? [15:06:47] doesn't seem like it [15:06:55] OK, I'm just going to add it to vendor [15:07:19] d'oh, but WHICH vendor [15:07:39] we have symfony in the root vendor dir and also in civicrm/packages [15:09:15] 06Fundraising-Backlog, 10FR-donorservices: Issue with Civi groups - https://phabricator.wikimedia.org/T366081 (10SHust) 03NEW [15:12:04] ah [15:13:13] well, i'm assuming that's where the deplay is [15:13:35] *delay [15:14:16] oh i can add some more logging in Tokens.php [15:21:17] ejegg: it looks like we get the slower processing for non-english templates [15:22:11] (03PS2) 10Ejegg: Log timings in TokenSmarty [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036685 [15:22:31] ah ok jgleeson [15:22:38] so some of that may be caching [15:22:46] it may be slower on the first of any particular language [15:23:24] and if they're mostly en, we're less likely to get multiples of any single translation in a one minute run [15:23:38] want to take a look at that latest logging patch? [15:24:01] so the last batch I looked at were swedish [15:24:02] I just added the logging directly to the subscribed functions rather than trying to mess with the symfony dispatcher code [15:24:14] ah, and did multiple ones have a problem? [15:24:17] for the preferred lang of EN we get no delay [15:24:19] in a single run? [15:24:27] but for Swedish, they all took 2s+ [15:24:40] interesting [15:25:20] actually that is more like 1-2s [15:25:47] looking [15:26:16] I'm gonna track down some of the earlier 10 second offenders and look at those [15:27:11] (03CR) 10Jgleeson: [C:03+2] "LGTM" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036685 (owner: 10Ejegg) [15:28:13] (03Merged) 10jenkins-bot: Log timings in TokenSmarty [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036685 (owner: 10Ejegg) [15:28:24] thanks! [15:28:38] CRM_Core_Smarty::singleton()->pushScope($smartyAssigns); [15:28:41] interesting [15:30:12] !log fundraising civicrm upgraded from 7e998894 to 4dd78bcc [15:30:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:33:23] jesus $listener($listener instanceof WrappedListener ? new LegacyEventProxy($event) : $event, $eventName, $this); [15:34:01] I think there was 24 listeners [15:34:04] wait, did I write Finishing for a starting run? [15:34:52] yes i did. that's confusing [15:34:55] lemme fix that [15:36:33] and there seem to be some early returns I'm not catching [15:39:19] oops [15:44:04] (03PS1) 10Ejegg: Fixes for parent logging patch [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036692 [15:47:02] ejegg: should we try setting that time limit to 50 to quiet down the failmail [15:47:24] ironically it is processing them faster but the cleanup must be longer than 5(7) seconds [15:47:52] ok, let's try 52 maybe? [15:47:59] sounds good [15:50:32] crap, the delay is picking back up [15:50:59] i guess if we're missing whole runs due to those overtime things that wouldn't help [15:51:04] ejegg: that is +2ed btw. looks like wiki bugs got kicked [15:51:11] ah thanks [15:52:22] !log fundraising civicrm upgraded from 4dd78bcc to 3fee95bc [15:52:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:52:41] (03Merged) 10jenkins-bot: Fixes for parent logging patch [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036692 (owner: 10Ejegg) [16:06:36] (03PS1) 10Damilare Adedoyin: Log payment details for audit [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1036697 [16:17:16] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Email: Civi to Acoustic Export: Job "MASTER SUPPRESSION LIST" failing every other day. - https://phabricator.wikimedia.org/T366059#9838573 (10KHaggard) p:05Triage→03Unbreak! [16:18:49] (03PS1) 10Jgleeson: Log preferred language in thank you email send action [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036698 [16:19:45] ejegg: ^^^ that's a small addition to make it easier to confirm the earlier thoughts about non-english templates [16:19:56] I tested it locally and it works [16:22:34] (03CR) 10Ejegg: [C:03+2] Log preferred language in thank you email send action [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036698 (owner: 10Jgleeson) [16:22:43] ok jgleeson [16:23:07] ah, so there was a longer run at the end of the hour with some of the longer delays [16:23:17] let's see if that has anything interesting [16:23:18] (03Merged) 10jenkins-bot: Log preferred language in thank you email send action [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036698 (owner: 10Jgleeson) [16:23:38] ty [16:24:41] thank_you_mail_send_smtp_frmx2001-20240528-155901.log [16:24:51] has some long delays on the last ones [16:25:06] ah, but not necessarily where we're looking right now [16:26:11] maybe it's not token-specific, maybe something else in civi core [16:27:03] there's like a 3 sec delay between the "2024-05-28 16:01:50,880 ERROR civicrm.wmf.INFO: thank_you: Attempting to send thank you mail for contribution ID [106440064]," and the 2024-05-28 16:01:53,538 ERROR civicrm.wmf.INFO: Creating new TokenProcessor [16:27:25] shoot, and is eileen out this week? [16:30:20] ejegg: is the TokenProcessor a wrapper timer? [16:30:55] I added a little script to count the diff between the lines and its showing the lag more clearly [16:30:57] 1567 ms 2024-05-28 16:28:30,807 ERROR civicrm.wmf.INFO: Creating new TokenProcessor [16:31:46] ah some don't have 'finishing' lines if that's what you mean [16:32:20] i guess those addRow and addMessage bits are not the issue [16:32:53] Hey fr-tech, let's repurpose the analytics hangout to get together and sync up on this TY email thing. I know folks are already working on it, but some face to face might help. [16:34:37] jgleeson: so that last email in the thank_you_mail_send_smtp_frmx2001-20240528-155901.log log [16:35:04] was 10s processing, and was for a donor in SE but with preferred language en [16:35:40] ejegg: this is what I mean about the TokenProcessor https://phabricator.wikimedia.org/P63456 [16:35:48] look how long that step takes [16:36:07] checking that log [16:36:41] ah yeah, so that would be a lot of stuff before we create it [16:36:53] ah ok [16:37:12] i think i was too hasty in deleting the first bit of logging i added [16:37:16] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fail Mail (civi1002) run-job: Get recipient data from Silverpop failed with code 1 - https://phabricator.wikimedia.org/T365544#9838689 (10KHaggard) Hi @Eileenmcnaughton I just unlocked your account, Damilare's account, and Wenjun's account. Let me know if... [16:39:07] should we log some cache calls? [16:39:41] dwisehaupt: can we see if there are some ungodly big cache keys being generated? [16:40:18] hmm, but if it's only degrading durirng a run i guess it's unlikely to be an external cache [16:40:59] ejegg: I don't see a 10seconder on that log [16:41:06] (03PS2) 10Damilare Adedoyin: Log payment details for Dlocal audit [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1036697 [16:41:21] /var/log/process-control/thank_you_mail_send_smtp_frmx2001/thank_you_mail_send_smtp_frmx2001-20240528-155901.log [16:41:23] ? [16:41:29] ejegg: sure, cache keys where? [16:41:31] ah i guess it was the one befor last [16:41:49] dwisehaupt: in the civi cache, i think redis [16:42:13] ejegg: I see a 6s [16:42:15] what ct_id? [16:43:06] jgleeson: at 16:01:39 [16:43:14] sure. [16:43:28] contribuition id 106440062 [16:43:29] also, the overview stats for civi redis are here: https://frmon.wikimedia.org/d/PkJ9u_PMk/redis?orgId=1&var-pool=civi&var-server=civi1002.frack.eqiad.wmnet&var-port=9005&from=now-7d&to=now [16:46:22] ejegg: that contact has a preferred language of spanish [16:46:25] I see it now [16:46:33] https://civicrm.wikimedia.org/civicrm/contact/view?reset=1&cid=63864398&selectedChild=summary [16:48:18] and here's the timings for that one ejegg https://phabricator.wikimedia.org/P63460 [16:48:46] the ms prefix is the time diff with the previous log line [16:49:09] the token processor setup took 3.5s [16:49:36] and 3s in the render space [16:51:06] the PII leak paranoia is high when making these pastes @_@ [16:53:20] ejegg: i'm not seeing anything standing out in first looks. stats look pretty consistent over the past 7 and 14 days. [16:53:24] i'll keep looking. [16:55:14] thanks dwisehaupt [16:55:32] i realize the external cache is actually unlikely here [16:56:27] looks like that batch time helped the faiimail [16:57:37] biggest key string is crm/metadata_5_74_beta1/CRM_Core_BAO_OptionValue_OptionGroupID_39_en_US at 386397 bytes. all the rest (5200 or so) average out to 3150.44 bytes each. [16:58:42] (as reported using --bigkeys with redis-cli) [17:10:10] thanks dwisehaupt [18:05:28] (the dang delay is going back up again) [18:10:00] 06Fundraising-Backlog, 10Web-Team-Backlog (Needs Prioritization (Tech)): Plan for Donate Wiki and Thank You Wiki Rollback and Redesign - https://phabricator.wikimedia.org/T361500#9839260 (10Jdlrobson) [18:18:39] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Civi-Dedupe: New Segmentation Framework not quite usable yet - https://phabricator.wikimedia.org/T363527#9839291 (10AKanji-WMF) [18:27:08] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Civi-Dedupe: How to deal with clearly bad email addresses in Civi - https://phabricator.wikimedia.org/T363946#9839336 (10AKanji-WMF) [18:28:12] 06Fundraising-Backlog: Large workplace giving companies not showing $ totals - https://phabricator.wikimedia.org/T364324#9839343 (10AKanji-WMF) 05Open→03Resolved [18:28:49] ok, so I got a big batch running locally and can see a slowdown happening [18:29:12] not sure exactly how I'll test with earlier versions of civi, but that's a start [18:35:00] (03PS1) 10Ejegg: WIP restore batch TY test functionality [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036730 [18:35:58] I guess I'll just try with earlier civi core tarballs and see if they give me db errors [18:36:27] will 'bisect' with 5.72 first [18:36:56] hmm, if i can find it that is [18:58:31] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Wikipedia-iOS-App-Backlog (iOS Release FY2023-24): [S] Strange API traffic pattern from WikipediaApp on payments.wikipedia.org - https://phabricator.wikimedia.org/T354575#9839553 (10Tsevener) 05Open→03Resolved [19:10:26] woot, got the batch test running under 5.72 [19:10:30] and no slowdown so far [19:10:37] awesome [19:10:55] ok, let's try 5.73 [19:14:51] anilk: did you have issue registering for the civicamp/sprint? I'm stuck on the registration form and when I try to submit by clicking either of the malformed buttons at the bottom, I just get redirected back to the earlier stage https://phabricator.wikimedia.org/F54558166 [19:18:45] ha the same form works fine in firefox [19:19:20] ok, no slowdown on 5.73.0 [19:19:25] gonna try 5.73.3 [19:21:59] no slowdown on 5.73.3 either [19:23:06] I guess to narrow down any further I do need to figure out how to run w/o a tarball [19:28:46] hi jgleeson|brb , no problems in firefox, I tested it again just now. Once I click on the "submit" button - nothing seems to happen (e.g. no loading indicators) - but after about 30 seconds I get directed to a confirmation screen [19:28:51] oh wow, actually, no slowdown on the latest 5.74 beta [19:28:58] let's try deploying that to prod! [19:29:56] (03PS1) 10Ejegg: Upstream Civi updates [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036738 [19:31:46] hmm, nothing in that change actually looks likely [19:35:34] (03Abandoned) 10Ejegg: Upstream Civi updates [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036738 (owner: 10Ejegg) [19:38:54] 14Fundraising Sprint: hammertime($touch_this=false), 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Allow DR to manually add a "Recurring Upgrade Declined" activity - https://phabricator.wikimedia.org/T362087#9839747 (10AKanji-WMF) [19:41:47] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog: Non-donor status missing records in our segmentation framework - https://phabricator.wikimedia.org/T363959#9839753 (10Eileenmcnaughton) a:03Eileenmcnaughton [19:41:49] 14Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog: Create new Benevity import - https://phabricator.wikimedia.org/T359219#9839754 (10XenoRyet) [19:41:58] 14Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog, 10FR-Civi-Dedupe, 07Unplanned-Sprint-Work: Sandra can't dedupe Name + address matching contacts (no email match) - https://phabricator.wikimedia.org/T353971#9839755 (10XenoRyet) [19:42:51] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog: Non-donor status missing records in our segmentation framework - https://phabricator.wikimedia.org/T363959#9839759 (10Eileenmcnaughton) @KHaggard the ones that have gone up over the past few days should have the new number / name [19:43:14] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2021: Spike - do practice standalong installs locally to get our heads around it - https://phabricator.wikimedia.org/T276395#9839761 (10XenoRyet) [20:15:17] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2021, and 2 others: Move a bunch of code out of wmf_civicrm.module - https://phabricator.wikimedia.org/T365415#9839822 (10AKanji-WMF) [20:21:07] !log revert to Smarty 2 revision changed from de15d068 to cdc89b59 [20:21:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:26:07] 06Fundraising-Backlog: Civi mismatch between contribution + Contribution Tracking details - https://phabricator.wikimedia.org/T366116 (10MBeat33) 03NEW [20:31:47] !log config revision changed from 6c4cd6c2 to 5b0b4d22 revert schedule [20:31:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:38:36] (03PS1) 10Ejegg: Revert "Log preferred language in thank you email send action" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036746 [20:38:36] (03PS1) 10Ejegg: Revert "Fixes for parent logging patch" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036747 [20:38:36] (03PS1) 10Ejegg: Revert "Log timings in TokenSmarty" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036748 [20:38:42] (03CR) 10Ejegg: [C:03+2] Revert "Log timings in TokenSmarty" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036748 (owner: 10Ejegg) [20:38:46] (03CR) 10Ejegg: [C:03+2] Revert "Fixes for parent logging patch" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036747 (owner: 10Ejegg) [20:38:50] (03CR) 10Ejegg: [C:03+2] Revert "Log preferred language in thank you email send action" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036746 (owner: 10Ejegg) [20:39:36] (03Merged) 10jenkins-bot: Revert "Log preferred language in thank you email send action" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036746 (owner: 10Ejegg) [20:39:36] (03Merged) 10jenkins-bot: Revert "Fixes for parent logging patch" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036747 (owner: 10Ejegg) [20:39:37] (03Merged) 10jenkins-bot: Revert "Log timings in TokenSmarty" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1036748 (owner: 10Ejegg) [21:03:28] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1036751 [21:03:31] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1036751 (owner: 10Ejegg) [21:04:52] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1036751 (owner: 10Ejegg) [21:05:19] !log payments-wiki upgraded from 2bfd247a to 0bed1814 [21:05:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:05:48] anilk: That just pushed out one small change to zh-hans and a translation of the new annual donation text ^^^ [21:05:56] not sure if that was the change pcoombe was waiting for [21:06:45] !log donorwiki upgraded from fa7de70f to 8ff002ef [21:06:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:07:05] thanks ejegg - I don't know how long his change to translatewiki would take to merge into code - it was a specific translation on the spanish donate form from the "ways to give page" - If we are able to deploy again tomorrow I was hoping that would catch it [21:07:38] ohh, on donatewiki, not on payments-wiki [21:07:58] sorry, that's on the main cluster deployment train [21:08:00] let's see [21:08:37] https://wikitech.wikimedia.org/wiki/Deployments/Train [21:08:55] should be out Thursday [21:09:33] err no, tomorrow [21:09:38] it's in group 1 [21:10:15] !log payments-wiki upgraded from 0bed1814 to 8ff002ef [21:10:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:10:40] ok, gotta go pick up my kid [21:15:57] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Delay on TY email sends - https://phabricator.wikimedia.org/T366120 (10AKanji-WMF) 03NEW [21:18:10] I've never made an UBN before https://phabricator.wikimedia.org/T366120 [21:18:31] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Delay on TY email sends - https://phabricator.wikimedia.org/T366120#9840122 (10AKanji-WMF) p:05Triage→03Unbreak! [21:18:49] fr-tech - feel free to add whatever notes to the link in the phab [21:25:17] RECOVERY - check_missing_thank_yous on frdb1005 is OK: OK missing_thank_yous=1195 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_missing_thank_yous [21:50:35] (03PS2) 10Eileen: WIP restore batch TY test functionality [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036730 (owner: 10Ejegg) [21:50:35] (03PS1) 10Eileen: Use QueueContext to log slow messages [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036757 [22:17:35] (03CR) 10CI reject: [V:04-1] Use QueueContext to log slow messages [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1036757 (owner: 10Eileen) [22:46:38] 14Fundraising Sprint: hammertime($touch_this=false), 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Allow DR to manually add a "Recurring Upgrade Declined" activity - https://phabricator.wikimedia.org/T362087#9840380 (10Ejegg) Sure, we should be able... [23:02:50] 06Fundraising-Backlog: Country ID error in Engage Organization import - https://phabricator.wikimedia.org/T366126 (10MDemosWMF) 03NEW [23:12:56] oh hah, i set phpstorm to analyze the snapshot while i was still appending to it [23:13:31] was wondering why it spent so long after it got to the end of the progress bar [23:13:36] hah, 37G [23:13:51] now that's some detailed profiling [23:21:52] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog: Non-donor status missing records in our segmentation framework - https://phabricator.wikimedia.org/T363959#9840493 (10KHaggard) Thanks. I'm meeting with Brian tomorrow to see if we can set a default field value for all non-donors to have "Non Donor" segm... [23:22:52] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog: Non-donor status missing records in our segmentation framework - https://phabricator.wikimedia.org/T363959#9840498 (10Eileenmcnaughton) great - thanks for updating me @KHaggard [23:23:33] huh, did it crash? [23:23:49] no tool window appears when it finished analyzing [23:23:59] even on a smaller dump [23:33:51] 06Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 10Thank-You-Page: Add payment method to thank you page URL - https://phabricator.wikimedia.org/T133072#9840509 (10Pcoombe) 05Open→03Resolved This was done some time ago