[04:24:45] 10Fundraising-Backlog: Acoustic suppressed emails 2023-10-19 - https://phabricator.wikimedia.org/T349358 (10Cstone) [12:32:11] (03CR) 10Jgleeson: [C: 03+2] "LGTM! It doesn't break anything when the donation is imported without the soon-to-follow corresponding CiviCRM updates in place." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/967266 (https://phabricator.wikimedia.org/T182039) (owner: 10Ejegg) [12:34:45] (03CR) 10Jgleeson: [C: 03+2] "Ejegg, I think we'll need a payments update for when we deploy this to bring in the new composer package." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/967266 (https://phabricator.wikimedia.org/T182039) (owner: 10Ejegg) [12:35:41] (03Merged) 10jenkins-bot: Add user-agent info to contribution_tracking [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/967266 (https://phabricator.wikimedia.org/T182039) (owner: 10Ejegg) [12:47:55] (03CR) 10Jgleeson: [C: 03+2] "It's working as intended. LGTM! thanks for the tests updates also here and over on the DI patch" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967271 (https://phabricator.wikimedia.org/T182039) (owner: 10Ejegg) [13:03:24] (03Merged) 10jenkins-bot: Save user-agent info to contribution_tracking table [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967271 (https://phabricator.wikimedia.org/T182039) (owner: 10Ejegg) [13:20:21] (03CR) 10Jgleeson: [C: 04-1] "This works! Thanks for the test also. I've asked if we can constantize some strings used in conditions and create tickets for those potent" [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/967250 (https://phabricator.wikimedia.org/T295726) (owner: 10Ejegg) [13:25:37] (03CR) 10Jgleeson: "Interesting. The behaviour you're replacing here could be used to solve the issue explained in Evelyn's note here https://wikimedia.slack." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/967238 (https://phabricator.wikimedia.org/T295726) (owner: 10Ejegg) [13:40:52] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/967454 [13:40:54] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/967454 (owner: 10Ejegg) [13:41:43] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/967454 (owner: 10Ejegg) [13:42:15] (03PS2) 10Ejegg: PayPal: Tag rather than discard GiveLively audit rows [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/967250 (https://phabricator.wikimedia.org/T295726) [13:42:46] (03CR) 10Ejegg: PayPal: Tag rather than discard GiveLively audit rows (032 comments) [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/967250 (https://phabricator.wikimedia.org/T295726) (owner: 10Ejegg) [13:43:17] (03CR) 10Ejegg: PayPal: Tag rather than discard GiveLively audit rows (032 comments) [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/967250 (https://phabricator.wikimedia.org/T295726) (owner: 10Ejegg) [13:51:17] 10Fundraising-Backlog: Acoustic suppressed emails 2023-10-19 - https://phabricator.wikimedia.org/T349358 (10Ejegg) @bsisolak I'm not sure if you're monitoring Phab, but we're wondering why Acoustic seems to have started sending 'Suppressed' events when we suppress donors via our scheduling process. Our Civi inte... [13:54:30] (03CR) 10Jgleeson: [C: 04-1] "This one looks good for the most part also. Just a few questions. thanks!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/967238 (https://phabricator.wikimedia.org/T295726) (owner: 10Ejegg) [14:08:48] !log fundraising civicrm upgraded from f11ad380 to e57425a9 [14:08:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:09:36] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/967459 [14:09:39] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/967459 (owner: 10Ejegg) [14:11:21] (03PS1) 10Ejegg: Update composer.lock with whichbrowser [core] (fundraising/REL1_39) - 10https://gerrit.wikimedia.org/r/967461 [14:11:31] (03CR) 10Ejegg: [C: 03+2] Update composer.lock with whichbrowser [core] (fundraising/REL1_39) - 10https://gerrit.wikimedia.org/r/967461 (owner: 10Ejegg) [14:19:28] (03Merged) 10jenkins-bot: Update composer.lock with whichbrowser [core] (fundraising/REL1_39) - 10https://gerrit.wikimedia.org/r/967461 (owner: 10Ejegg) [14:25:32] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/967459 (owner: 10Ejegg) [14:49:04] !log payments-wiki upgraded from 87cda414 to 7575f0e6 [14:49:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:53:34] woohoo, we're getting useragent info in contribution_tracking! [14:53:41] Now just to add it to the displays [14:56:36] 10Fundraising-Backlog: Greece: Set Adyen with post-payment monthly convert as Default - https://phabricator.wikimedia.org/T349122 (10AKanji-WMF) Thanks @HNordeenWMF - am having trouble getting into the Asana task, could you please confirm timing? [15:05:13] 10Fundraising-Backlog: Acoustic suppressed emails 2023-10-19 - https://phabricator.wikimedia.org/T349358 (10bsisolak) This was related to something @KHaggard sent me an email on, and that a large suppression list was added at the time of send. You have been getting suppressed events from Acoustic for all time,... [15:10:28] 10Fundraising Sprint Dinosaur Cookies co-existed with Gingerbread People, 10Fundraising Sprint This LAN is your LAN, 10Fundraising-Backlog, 10MW-1.42-notes (1.42.0-wmf.2; 2023-10-24): investigate: Record user agent in contribution_tracking - https://phabricator.wikimedia.org/T182039 (10Ejegg) This is deplo... [15:36:21] 10Fundraising-Backlog: Suppress Paypal Grant CID to avoid duplicate db recording - https://phabricator.wikimedia.org/T349392 (10EMartin) [15:52:42] 10Fundraising-Backlog: Acoustic suppressed emails 2023-10-19 - https://phabricator.wikimedia.org/T349358 (10KHaggard) p:05Triage→03High I'll also add that the reason why this is the first time we're seeing this is that we usually don't rely much on suppression lists because we handle everything with queries,... [16:12:04] 10Wikimedia-Fundraising-Banners, 10Wikipedia-Android-App-Backlog (Android Release - FY2023-24), 10Wikipedia-iOS-App-Backlog (Design), 10iOS Release FY2023-24: JSON for new app fundraising banners (multilingual, Italy and Netherlands) - https://phabricator.wikimedia.org/T348278 (10Sharvaniharan) 05Open→0... [16:25:51] 10Fundraising-Backlog: Acoustic suppressed emails 2023-10-19 - https://phabricator.wikimedia.org/T349358 (10bsisolak) More context. There are three types of suppressions (System, Org, Mailing). System suppressions are domains that are suppressed on all of Acoustic as they are not real, spam trap, generic garbage... [16:41:11] 10Fundraising-Backlog: Suppress Paypal Grant CID to avoid duplicate db recording - https://phabricator.wikimedia.org/T349392 (10Ejegg) Looks like we can just block incoming donations based on the email address - pretty sure we can put that in code or public config since it's an @paypal.com email that they seem t... [17:13:14] (03PS1) 10Cstone: Remove Supressed from unsubscribes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967481 (https://phabricator.wikimedia.org/T349358) [17:14:05] ejegg: would that just be all we have to do to stop importing supressed ^ [17:15:40] 10Fundraising-Backlog, 10Patch-For-Review: Acoustic suppressed emails 2023-10-19 - https://phabricator.wikimedia.org/T349358 (10Cstone) @bsisolak Currently 'Opt Out', 'Reply Abuse', 'Suppressed' are all marked unsubsrcibes, the above patch is removing Suppressed from that list [17:29:00] (03CR) 10CI reject: [V: 04-1] Remove Supressed from unsubscribes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967481 (https://phabricator.wikimedia.org/T349358) (owner: 10Cstone) [17:33:25] oh makeScientists hah [17:34:32] heh, I think that's all (plus the test fix) [17:34:58] im guessing issac is supressed but hunting for where that is setup to confirm [17:36:11] omg ive been spelling that wrong forever [17:36:13] two as??? [17:36:39] haha, yeah, it's weird [17:36:48] aardvark newton [17:40:19] (03PS2) 10Cstone: Remove Supressed from unsubscribes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967481 (https://phabricator.wikimedia.org/T349358) [17:42:24] cstone: so for the revert, maybe it'll be faster if we pass in the 'tables', 'log_date', and 'interval' parameters [17:43:52] huh, _civicrm_api3_logging_revert_spec says interval is required when using log_date, but I don't see interval being used [17:44:48] seems to default to 10 sec down in CRM_Logging_Differ [17:47:15] so yeah, sending the tables and log_date for each call would filter the queries more [17:48:31] or hmm, can we just use the logging.get command to figure out whose values actually changed? [17:49:08] if we can tell that everyone actually had is_opt_out=0 before this update, that would make it easier to just do a big undo in the db [17:49:53] ehh, though it's still 32k API calls just to check [17:57:16] ejegg: when estimating time things would take with long time is that like hours or days? [17:57:23] good question [17:57:29] im ok watching a longer running script if it cleans everything up nicely [17:58:01] we can try something on staging [17:58:07] 10Fundraising-Backlog, 10fundraising-tech-ops, 10Patch-For-Review: Set up Community CiviCRM instance. - https://phabricator.wikimedia.org/T343486 (10Dwisehaupt) Noting for posterity from [[ https://wikimedia.slack.com/archives/D05V6HQNERG/p1696448929560149 | slack conversation ]], the instance will be hosted... [17:58:24] but of course there it's not conflicting with the rest of the processes [17:58:42] hmm searchkit has a unhold addresses would that revert [17:59:20] 10fundraising-tech-ops, 10Infrastructure-Foundations, 10SRE, 10vm-requests: eqiad: 1 VM requested for community-crm - https://phabricator.wikimedia.org/T349402 (10Dwisehaupt) [17:59:22] so the code sets the address to on_hold but it also sets is_opt_out to 1 on the contact [17:59:38] and I'm not sure that ALL of those contacts had is_opt_out=0 beforehand [18:00:04] I mean, probably so since they were added to the suppression list at Acoustic [18:00:15] and presumably the didn't need to add anyone that was already on the MSL [18:01:04] so we can make a table with the log_conn_id and the log_date for all those changes [18:01:20] then run an update that does a batch of 100 and times it [18:01:45] if i can get a search kit to match everything we could do both of those from search kit too [18:02:19] oh, does search kit have log reverts? [18:02:31] hmmm i dont know [18:02:51] what is that vs just changing the status again? [18:03:01] it's that uncertainty about whether any of them were already opted out beforehand [18:03:11] (or on hold) [18:03:47] hmm [18:04:02] so the log revert will always make them end up where they were before the change [18:04:50] we could probably do a DB join to the log tables to check if there was anyone that was already opted out [18:05:19] want to hop on a call and look at the data together? [18:05:27] sure [18:05:36] today's standup? [18:05:49] yeah lemme just get a drink [18:49:03] (03PS1) 10Damilare Adedoyin: Add Activity to contact profile when snoozed. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967509 (https://phabricator.wikimedia.org/T349277) [18:52:46] (03PS2) 10Damilare Adedoyin: Handle Adyen refund IPNs for non-USD donations [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967264 (https://phabricator.wikimedia.org/T347422) [18:54:16] (03CR) 10Damilare Adedoyin: "Thanks ejegg for the CR, I've added the update to the original currency as discussed." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967264 (https://phabricator.wikimedia.org/T347422) (owner: 10Damilare Adedoyin) [18:54:34] (03PS1) 10Ejegg: Test reverting a batch of 100 unsubscribes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967511 (https://phabricator.wikimedia.org/T349358) [18:58:11] oh derp cstone i've got contact_activity not activity_contact in my table list [18:58:21] hah [18:58:40] i realize i can run just a single one on the command line to test that it does the right thing [18:58:51] oh like the revert? [18:58:55] yah [18:58:58] nice [18:59:06] hmm, if i can send that table list in ok [18:59:31] does just comma-separated work? [19:01:11] hmm getting some error locally [19:02:05] (03PS2) 10Ejegg: Test reverting a batch of 100 unsubscribes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967511 (https://phabricator.wikimedia.org/T349358) [19:02:27] do the upgrade logs go somewhere else [19:03:25] just giving me a helpful CIVICRM api error [19:04:35] bah, on the commandline I'm getting DB Error: no such table [19:04:57] with the few different ways of passing tables I've tried [19:05:14] does it work passing none? [19:05:49] yep, just did [19:05:59] took 1.5 sec, but that was the whole drush command [19:06:14] so like .75 sec of that was probably bootstrapping [19:06:42] lemme see if I can debug through that upgrade locally [19:06:55] hah im attempting that and debugger is not behaving [19:07:36] oh, but of course there won't be any actual matching data locally [19:07:59] it should still make the table though just wouldnt have anything? [19:08:01] guess i could fake something up with just an insert into the mailing data table and running the job [19:08:07] hmm, I think it should? [19:08:20] but I'm not sure now [19:08:31] since it won't have any real data to infer columns from [19:08:34] im just getting CIVICRM api error trying to figure out where thats coming from [19:08:43] any backtrace? [19:08:46] nope [19:09:00] ok, lemme run here [19:10:06] (03PS3) 10Ejegg: Test reverting a batch of 100 unsubscribes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967511 (https://phabricator.wikimedia.org/T349358) [19:11:39] huh, no error, but... no evidence it actually did anything either [19:11:45] let's see, is the table there? [19:12:11] narp [19:12:33] I did drush @wmff cvapi extension.upgrade [19:12:39] maybe if I try from the UI? [19:14:05] I was trying the same thing [19:14:17] Er from drush [19:14:31] weird, let me see what civi thinks the version is [19:14:52] Yeah I was just going to check mine [19:16:06] k, i guess i had run a different update with that same number before, probs when reviewing Eileen's code [19:16:12] so now I get the same API Error [19:16:16] what table is that in again [19:16:17] oh, I need to turn on cli debugging [19:16:34] select * from civicrm_extension where name like '%wmf%'; [19:17:13] my debugger just hangs im not sure wahts going on there [19:17:24] ohhh some cv randomness? [19:19:18] i think the error is coming from the table create [19:19:19] I have some hatchet-patch i cherry-pick locally when i need to debug drush [19:19:23] ahhh ok [19:19:32] lemme try doing that in SQL locally where there are no results [19:19:45] i commented everything else out and still get it [19:21:03] is the table created? [19:21:13] i dont see it being created [19:21:18] It seemed to have gotten created before that error was thrown locally for me [19:21:24] ooh okay [19:21:32] when I tried to create it in sql it said Table 'T349358' already exists [19:21:32] T349358: Acoustic suppressed emails 2023-10-19 - https://phabricator.wikimedia.org/T349358 [19:21:45] maybe the index? [19:22:42] lemme see [19:22:45] where would the table appear drupal or civi? [19:24:09] civi [19:25:07] I'll just do a normal table creation with an autoincrement ID, then select those rows into it [19:26:16] when i manually pasted in the sql it appeared [19:26:53] oh huh [19:27:05] ok, let me try with a more normal create and insert [19:27:20] ive gone a tab too far sorry laptop [19:34:11] (03PS4) 10Ejegg: Test reverting a batch of 100 unsubscribes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967511 (https://phabricator.wikimedia.org/T349358) [19:34:24] ok, currently trying ^^^ locally [19:35:53] seems to work, though of course there is no data to revert [19:36:13] guess the executeQuery doesn't like alter table or something [19:38:42] didn't see those log lines anywhere [19:38:47] maybe in configandlog? [19:38:56] ah i didnt look there yet ejegg my syslog was nicely empty [19:39:23] yah I think the \Civi::log('wmf') goes to syslog [19:39:44] ahh hah [19:40:08] bah, my configandlog locally has nothing since Jun 28??? [19:41:05] i'mma just swap to 'wmf' logging [19:41:32] im still getting that api error but maybe its me then if you arent seeing it [19:44:31] derp, logging disappears there too [19:44:54] oh hmm, is the table already there? [19:45:49] no [19:45:50] actually, i should have gotten an error trying to re-run [19:45:53] hmm hmm [19:46:11] and you say the sql works for you on the mysql cli? [19:46:24] the original sql did yeah lemme try your chunked one [19:47:34] chunked sql works too on the cli [19:49:13] ok, it's working for me now, i hadn't actually reset the module version when i thought I did the last time [19:49:17] lemme push one more up [19:49:37] (03PS5) 10Ejegg: Test reverting a batch of 100 unsubscribes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967511 (https://phabricator.wikimedia.org/T349358) [19:49:39] ok yeah if it works for you my local is probably messed up [19:49:46] and the fact i can just run the sql by itself [19:49:56] so now I can see the log messages in the syslog too [19:50:01] nice! [19:50:09] where is config and log locally [19:50:41] 10Fundraising-Backlog: Unexpected $0 auths on donor statements - https://phabricator.wikimedia.org/T348715 (10EMartin) We have gone to the issuing banks to see if we can get more info about the origin of these $0 auths. [19:51:10] so i'm just looking at the top-level logs/civicrm-syslog file now [19:51:20] the configandlog still doesn't have anything from this month for me [19:53:40] ok my local is definately broken [19:54:54] want to hop back on that call? [19:55:01] its not your patch though ejegg sure [20:00:33] (03CR) 10Cstone: [C: 03+2] Test reverting a batch of 100 unsubscribes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967511 (https://phabricator.wikimedia.org/T349358) (owner: 10Ejegg) [20:01:53] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10Patch-For-Review: Acoustic suppressed emails 2023-10-19 - https://phabricator.wikimedia.org/T349358 (10AKanji-WMF) [20:07:48] (03PS3) 10Ejegg: Remove Supressed from unsubscribes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967481 (https://phabricator.wikimedia.org/T349358) (owner: 10Cstone) [20:07:53] (03CR) 10Ejegg: [C: 03+2] Remove Supressed from unsubscribes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967481 (https://phabricator.wikimedia.org/T349358) (owner: 10Cstone) [20:10:54] (03CR) 10Ejegg: [V: 03+2 C: 03+2] Remove Supressed from unsubscribes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967481 (https://phabricator.wikimedia.org/T349358) (owner: 10Cstone) [20:19:12] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/967528 [20:19:16] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/967528 (owner: 10Ejegg) [20:20:07] (03CR) 10Ejegg: [V: 03+2 C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/967528 (owner: 10Ejegg) [20:20:50] !log fundraising civicrm upgraded from e57425a9 to 1263a91b [20:20:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:24:16] ejegg: I'm going to turn the unsubscribe job back on [20:27:50] cstone sounds good. so the library's closing in half an hour, and I want to get some books for Tavi. [20:28:24] lemme grab those and see if I can find a place to work besides home so I can actually concentrate and get this done [20:28:33] i'll be back on in half an hour or so [20:28:33] ok sounds good ejegg thanks for all the help [20:29:44] anyway, I think we want civicrm_api3('Logging', 'revert', $params); [20:29:53] in case you want to make that fix [20:30:04] yeah I'll have a go at it [20:37:16] (03CR) 10Wfan: [C: 03+2] "LGTM, thanks" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967264 (https://phabricator.wikimedia.org/T347422) (owner: 10Damilare Adedoyin) [20:53:17] (03Merged) 10jenkins-bot: Handle Adyen refund IPNs for non-USD donations [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967264 (https://phabricator.wikimedia.org/T347422) (owner: 10Damilare Adedoyin) [21:10:07] cstone: ok, I'm back. Did you end up getting some food? [21:10:19] ok hey ejegg no i was testing haha [21:10:31] i just reverted my test locally but am seeing Notice: Undefined index: addressee_display in include() (line 69 of /srv/civicrm-buildkit/app/private/wmff/default/civicrm/templates_c/en_US/%%77/77A/77AF58B9%%CommunicationPreferences.tpl.php). [21:11:28] in civi after the revert [21:11:59] bah, I did see some places where addressee_display was touched in the contact save [21:12:25] but we should be able to revert still [21:13:03] that was with your change did we want to figure out batching with this one or just do this 100 fix first [21:13:42] (03PS1) 10Ejegg: Break up table create and batch run [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967533 (https://phabricator.wikimedia.org/T349358) [21:14:05] Hmm, I still want to know how long 100 takes when running in code [21:15:20] ok [21:15:46] but yeah besides that notice it looks good, the activity is gone, the email isn't red [21:15:54] So I feel like we see a lot of undefined index notices in templates [21:16:36] and those are mostly not real problems [21:16:40] yeah [21:16:42] like I think we just see them in dev [21:16:48] and they're hidden in prod [21:16:50] yeah makes sense [21:18:46] 10Fundraising-Backlog: Renew Your API Certificate - Email from tle@wikimedia.org - https://phabricator.wikimedia.org/T348725 (10AKanji-WMF) Needs to be renewed before September 2024 [21:20:46] bah lost my omnimail window hah [21:26:56] hmm ejegg my version which is the same as yours worked fine but with your patch its not deleting it double checking my data though its probably that [21:27:45] (03PS2) 10Ejegg: PayPal: Tag rather than discard GiveLively IPNs [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/967238 (https://phabricator.wikimedia.org/T295726) [21:28:12] (03CR) 10Ejegg: "Thanks for the review jgleeson. Problems addressed in PS2" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/967238 (https://phabricator.wikimedia.org/T295726) (owner: 10Ejegg) [21:29:38] ah had the wrong date [21:30:03] perfect [21:30:10] :) [21:30:14] (03CR) 10Cstone: [C: 03+2] "works for me locally!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967533 (https://phabricator.wikimedia.org/T349358) (owner: 10Ejegg) [21:30:27] (03CR) 10Ejegg: [V: 03+2] Break up table create and batch run [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967533 (https://phabricator.wikimedia.org/T349358) (owner: 10Ejegg) [21:31:24] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/967534 [21:31:26] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/967534 (owner: 10Ejegg) [21:31:33] (03CR) 10Ejegg: [V: 03+2 C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/967534 (owner: 10Ejegg) [21:33:59] !log fundraising civicrm upgraded from 1263a91b to ca081c11 [21:34:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:34:03] after all this still cant spell suppressed [21:34:04] why two ps [21:34:09] k, let's try this again [21:34:11] oh no [21:34:23] oh no?? [21:34:28] ah, but no bearing on the code, right? [21:34:40] oh yeah sorry [21:34:42] not on the code [21:34:50] making a searchkit for katie haha [21:35:35] upgrading again [21:37:46] andddd CIVICRM api error [21:37:49] d'oh [21:37:56] looking for logs [21:38:28] whattt [21:38:35] table is there [21:38:37] the one time i didn't see that! [21:38:37] haha [21:38:54] has the 32178 rows in it [21:39:09] all is_processed=0; [21:39:22] hmm [21:40:03] so i never saw the Starting revert log line [21:40:05] did it log any of your lines? [21:40:15] on the console anyway [21:40:18] looking in syslog [21:40:31] 2023-10-20T21:29:40+00:00 8cb471eac0d3 civicrm.wmf[3364]: civicrm.wmf.INFO: Starting revert of 100 unsubscribes at 1697837380 [21:40:31] 2023-10-20T21:29:40+00:00 8cb471eac0d3 civicrm.wmf[3364]: civicrm.wmf.INFO: Reverted 1 unsubscribes in 0 seconds [21:40:36] i saw that in syslog locally [21:40:48] huh, that looks good then [21:41:31] nope, it didn't log that in syslog either [21:41:47] could it not see the custom table somehow? [21:43:24] ohhh [21:43:32] schema_version=4395 [21:43:37] it's trying to make the table again [21:43:42] lemme fudge that [21:43:48] oooh [21:43:59] i guess i had hacked that number locally cause I ran it so many times [21:44:29] k, running again [21:44:58] civicrm.wmf.INFO: Reverted 100 unsubscribes in 5 seconds [21:45:01] woot [21:45:03] woo!!!!!! [21:45:29] so like half an hour for the whole lot of 'em [21:45:39] not bad [21:45:45] what do you say we just turn queues off and run through 'em all? [21:46:01] not like we're worried about skipping the acoustic export tonight [21:46:02] they did just put a 5% us test live [21:46:05] ahhh [21:47:00] I guess let's ask folks if they're actively monitoring, or if it's ok to get a bit of lag on the numbers [21:47:21] but first, I'mma check on a few of the contacts in that batch of 100 [21:47:22] but yeah I was going to check the load too [21:48:22] for the 5% [21:48:30] ill ask banner peeps though [21:48:37] thanks! [21:52:36] ok, spot-checked 8 of them and all look like they reverted correctly [21:52:40] nice! [21:54:00] if we don't get a response in like 5 mins ejegg I bet its safe enough to turn queues off as in no one around actively monitoring wise [21:54:14] havent seen any sshs by peeps either [21:57:37] k, cool [21:57:50] yeah i say lets just go ahead and do it [21:59:07] (03PS1) 10Ejegg: Revert the rest of the erroneous unsubscribes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967537 (https://phabricator.wikimedia.org/T349358) [21:59:27] i should actually smoke test this one locally... [21:59:55] grr ide adding spaces [22:00:55] (03PS2) 10Ejegg: Revert the rest of the erroneous unsubscribes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967537 (https://phabricator.wikimedia.org/T349358) [22:03:45] testing with my 1 row hah [22:04:43] ejegg: are you going to hack the upgrade number again? [22:05:10] ooh it increased one sorry hah [22:05:33] oh hah, i need to revert your patch locally for it to actually unsubscribe that suppressed row [22:06:28] hah I changed my test data to just be Opt Out [22:07:00] ohh, and you used the log_conn_id from your manual change? [22:08:31] yeah i update that and the date [22:09:08] so it get unsubscribed then undo that [22:09:41] new version looks good to me ejegg [22:09:42] 2023-10-20T22:08:18+00:00 8cb471eac0d3 civicrm.wmf[3479]: civicrm.wmf.INFO: Starting revert of 1000 unsubscribes at 1697839698 [22:09:42] 2023-10-20T22:08:18+00:00 8cb471eac0d3 civicrm.wmf[3479]: civicrm.wmf.INFO: Reverted 1 unsubscribes in 0 seconds [22:10:11] ok, i seem to have gotten it to work by adding the mailing_data row and running drush -vv cvapi omnirecipient.process_unsubscribesdrush -vv cvapi omnirecipient.process_unsubscribes [22:10:21] (with your patch reverted) [22:10:25] ahh okay [22:10:37] just gonna roll back my schema version and try the full fix [22:11:28] shoot, mosquitoes found me [22:11:44] and it's getting dark and slightly chilly in this park [22:12:43] huh, maybe I had a date wrong [22:12:45] civicrm.wmf.INFO: Reverted 0 unsubscribes in 0 seconds [22:13:06] the date is being weird its like it copies the wrong one from mysql workbench [22:13:10] gonna hack that date and see if it works [22:13:18] i have had to copy it twice the last 2 times [22:14:09] also official okay to pause queues [22:14:18] great!@ [22:14:31] freaky, I swear I inserted a different recipient_action_datetime [22:14:40] looks like it's been TZ-corrected [22:17:26] ohhhh my activity_type_id for silverpop unsubscribe is not 99 locally [22:17:50] just scrubbing that for the moment [22:18:26] is that one of the ones we can fix i think eil een was adding patches like that to dev to get them to match [22:18:55] maybe [22:19:35] but if it mattered to make it work on dev we could just join to the activity type options and go by name [22:19:49] i''m just deleting that bit of the join for now [22:20:15] bah, rewound the version too far and something borked [22:20:22] hope it didn't really screw up the db [22:20:33] (still all local) [22:20:45] nope, no problems at all [22:20:57] and it reverted 1 unsub in 0 sec [22:21:00] woo! [22:21:03] ok, I'm happy with that then [22:21:14] ok looks good to me too! [22:21:16] just gonna flee from these little flying bloodsuckers [22:21:30] (03CR) 10Cstone: [C: 03+2] "looks good!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967537 (https://phabricator.wikimedia.org/T349358) (owner: 10Ejegg) [22:21:30] and try to work from chaos house [22:21:35] brb [22:33:43] it's a joyful chaos [22:33:51] but I've gotta help with some stuff here [22:34:15] cstone: do you want to take over? [22:34:37] I'll be pingable just in case [22:34:46] I can ejegg thanks for all the work on this [22:34:57] :) :) always interesting [22:36:39] (03Merged) 10jenkins-bot: Revert the rest of the erroneous unsubscribes [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/967537 (https://phabricator.wikimedia.org/T349358) (owner: 10Ejegg) [22:42:40] (03PS1) 10Cstone: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/967538 [22:42:54] (03CR) 10Cstone: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/967538 (owner: 10Cstone) [22:43:41] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/967538 (owner: 10Cstone) [22:47:08] !log civicrm upgraded from ca081c11 to 8e8ffec0 [22:47:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:49:27] (03PS10) 10Wfan: Add declineMonthlyConvert as param for post Monthly Convert donation api [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/963970 (https://phabricator.wikimedia.org/T347606) [22:53:46] ejegg|away: sorry but not entirely sure how to actually run the disable script to get process control off [22:57:45] oh oh nevermind!! [23:09:10] queues are off and update is chugging along [23:35:16] * greg-g makes train noises [23:37:23] ok update is done [23:47:25] queues back on [23:49:06] silverpop back on [23:49:10] cool. thanks cstone [23:52:50] cstone++