[00:18:49] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Civi DB Error: constraint violation detected from civicrm_mailing_event_delivered - https://phabricator.wikimedia.org/T319445 (10AnnWF) [00:19:59] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Civi DB Error: constraint violation detected from civicrm_mailing_event_delivered from recurring_queue_consume log - https://phabricator.wikimedia.org/T319445 (10AnnWF) [01:03:43] (03PS1) 10Ejegg: WIP script to fill missing email greetings [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/838948 (https://phabricator.wikimedia.org/T308181) [02:15:56] (03PS1) 10Ejegg: Sync new custom groups for MG/Endowment appeals to dev [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/838980 (https://phabricator.wikimedia.org/T318529) [02:27:27] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10donate.wikimedia.org: Broken links from Waystogive? - https://phabricator.wikimedia.org/T315268 (10Ejegg) Still seeing a couple of these per day, but I notice some have some pretty strange combinations of uselang, country, and currency. I'll ask for a... [02:49:01] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Civi merged two CID but did not save/include an active recurring donation - https://phabricator.wikimedia.org/T319073 (10Cstone) a:03Cstone [05:19:26] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_39) - 10https://gerrit.wikimedia.org/r/839139 (owner: 10L10n-bot) [08:05:25] 10Fundraising Sprint NaN is a Number, 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising Sprint Quarkery, and 4 others: Email Greeting Field in Civi - blank for some donors - https://phabricator.wikimedia.org/T308181 (10RLewis) @Ejegg - I think that sounds like... [08:12:29] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Adyen:2) 2 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [08:17:29] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [10:10:07] (03CR) 10Awight: [C: 03+2] Optimize regular expression code in PHP and JavaScript (032 comments) [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/683619 (owner: 10Thiemo Kreuz (WMDE)) [10:30:22] (03Merged) 10jenkins-bot: Optimize regular expression code in PHP and JavaScript [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/683619 (owner: 10Thiemo Kreuz (WMDE)) [11:06:29] (03CR) 10Thiemo Kreuz (WMDE): Optimize regular expression code in PHP and JavaScript (031 comment) [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/683619 (owner: 10Thiemo Kreuz (WMDE)) [11:29:36] (03CR) 10Awight: [C: 03+2] Optimize regular expression code in PHP and JavaScript (031 comment) [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/683619 (owner: 10Thiemo Kreuz (WMDE)) [11:34:30] 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Performance-Team, 10Product-Analytics, and 2 others: CentralNotice severely impacts CLS score - https://phabricator.wikimedia.org/T318046 (10Fuzzy) According to the Page Experience chart, the problem started August 30th. I cannot say when the c... [13:10:59] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10donate.wikimedia.org: Broken links from Waystogive? - https://phabricator.wikimedia.org/T315268 (10Pcoombe) Thanks Elliott. Worth noting that I removed these PayPal links from Ways to Give in English on 15 September, although they are still present in... [13:17:58] 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Performance-Team, 10Product-Analytics, and 2 others: CentralNotice severely impacts CLS score - https://phabricator.wikimedia.org/T318046 (10Pcoombe) The [sound logo campaign](https://meta.wikimedia.org/w/index.php?title=Special:CentralNotice&s... [13:52:04] 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Performance-Team, 10Product-Analytics, and 2 others: CentralNotice severely impacts CLS score - https://phabricator.wikimedia.org/T318046 (10Fuzzy) There might be other causes which started earlier. Still, according to the Lighthouse analysis,... [16:43:51] (03CR) 10Abijeet Patro: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_39) - 10https://gerrit.wikimedia.org/r/839139 (owner: 10L10n-bot) [17:49:46] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Civi merged two CID but did not save/include an active recurring donation - https://phabricator.wikimedia.org/T319073 (10Cstone) Hi @SHust this is a weird one! I think this was caused by a deadlock at the time of the dedupe happening... [18:09:26] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Civi merged two CID but did not save/include an active recurring donation - https://phabricator.wikimedia.org/T319073 (10SHust) Thanks, @Cstone, will do! [19:27:17] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: adyen_transaction_failure 119 [=10], check_endpoints_critical (Adyen:1) 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [19:32:16] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [19:47:57] (03PS1) 10Ejegg: Stop sending unrectifiable orphans to damaged [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/839681 [19:48:12] ok cstone I think ^^^ should do it [20:05:57] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10donate.wikimedia.org: Broken links from Waystogive? - https://phabricator.wikimedia.org/T315268 (10AnnWF) Based on the error message here we have both [payment_method] => and [amp;payment_method] => paypal , which suspect the root cause is that we do... [20:26:17] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10donate.wikimedia.org: Broken links from Waystogive? - https://phabricator.wikimedia.org/T315268 (10Ejegg) @AnnWF there are still some in the logs for October. It looks like the querystring is being escaped for some reason, so the &s between parameters... [21:09:35] (03PS1) 10Ejegg: Quiet down ChargebackReversed failmail [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/839703 (https://phabricator.wikimedia.org/T318175) [21:18:10] thanks for those ejegg I can test now [21:19:56] or my debugger completely stopped working why [21:39:22] arrrgh [21:39:35] cstone: oh are you on the new OS version of payments? [21:39:54] hmm I dont think so [21:40:15] would rebuilding everything get it? [21:40:28] yep, if you have the fundraising-dev repo up to date [21:40:43] but that would only affect debugging on the payments container [21:40:47] ok ill just redo everything way quicker than trying to debug this [21:40:52] yeah only that is broken [21:40:58] oh ok [21:41:46] so after the rebuild you still will probs have to edit config/payments/xdebug* and set start_with_request to yes [21:41:51] and maybe set idekey [21:43:40] does it only run batchprocessor step from the orphan rectifier though? [21:43:59] cstone: ohhhh right [21:44:21] yeah you'll need to run that old-school orphan slayer then [21:44:28] lessee what the commandline is [21:44:33] yeah thats fine im not worried about debugging through that [21:44:59] (and of course that runs it via donation-interface-the-library) [21:45:21] (03CR) 10Cstone: [C: 03+2] "Thanks for this!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/839681 (owner: 10Ejegg) [21:45:34] thank you cstone ! [21:45:44] thank you! hah other one too [21:46:10] (03CR) 10Cstone: [C: 03+2] "Thanks for this too!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/839703 (https://phabricator.wikimedia.org/T318175) (owner: 10Ejegg) [21:46:39] (03Merged) 10jenkins-bot: Quiet down ChargebackReversed failmail [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/839703 (https://phabricator.wikimedia.org/T318175) (owner: 10Ejegg) [21:49:18] (03Merged) 10jenkins-bot: Stop sending unrectifiable orphans to damaged [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/839681 (owner: 10Ejegg) [21:58:30] well I guess I'll deploy those [21:58:39] and probably do a manual cleanup of that damaged table [21:58:53] oops, gotta help with dinner first [21:59:50] (03PS7) 10Ejegg: Create Base API class for PayPal EC [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/837669 (https://phabricator.wikimedia.org/T318884) (owner: 10Jgleeson) [21:59:52] (03PS1) 10Ejegg: Rename IPN validator class, special-case validator [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/839730 (https://phabricator.wikimedia.org/T318884) [22:00:16] (03CR) 10CI reject: [V: 04-1] Create Base API class for PayPal EC [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/837669 (https://phabricator.wikimedia.org/T318884) (owner: 10Jgleeson) [22:00:19] (03CR) 10CI reject: [V: 04-1] Rename IPN validator class, special-case validator [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/839730 (https://phabricator.wikimedia.org/T318884) (owner: 10Ejegg) [22:07:27] (03PS2) 10Ejegg: Rename IPN validator class, special-case validator [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/839730 (https://phabricator.wikimedia.org/T318884) [22:07:29] (03PS8) 10Ejegg: Create Base API class for PayPal EC [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/837669 (https://phabricator.wikimedia.org/T318884) (owner: 10Jgleeson) [22:07:50] (03CR) 10CI reject: [V: 04-1] Rename IPN validator class, special-case validator [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/839730 (https://phabricator.wikimedia.org/T318884) (owner: 10Ejegg) [22:07:59] (03CR) 10CI reject: [V: 04-1] Create Base API class for PayPal EC [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/837669 (https://phabricator.wikimedia.org/T318884) (owner: 10Jgleeson) [23:37:08] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Donor with over 28 charges in Civi after single donation was made - https://phabricator.wikimedia.org/T319203 (10Cstone) We looked into this one and thought about it a lot, the logs show that they only started the flow once and then...