[03:34:20] <wikibugs>	 (03PS6) 10Cstone: WIP: Adding PaymentAccountingReport to the files the adyen audit looks at [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/831159 (https://phabricator.wikimedia.org/T314753)
[03:34:45] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] WIP: Adding PaymentAccountingReport to the files the adyen audit looks at [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/831159 (https://phabricator.wikimedia.org/T314753) (owner: 10Cstone)
[04:58:52] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_37) - 10https://gerrit.wikimedia.org/r/831695 (owner: 10L10n-bot)
[06:41:33] <wikibugs>	 (03CR) 10Abijeet Patro: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_37) - 10https://gerrit.wikimedia.org/r/831695 (owner: 10L10n-bot)
[12:34:10] <wikibugs>	 10fundraising-tech-ops, 10Infrastructure-Foundations, 10Mail: Investigate in-house DMARC analysis tool options - https://phabricator.wikimedia.org/T317443 (10Jgreen)
[16:06:33] <wikibugs>	 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising-Backlog, 10Patch-For-Review: Allow PayPal donations from CN - https://phabricator.wikimedia.org/T316703 (10Ejegg) 05Open→03Resolved
[16:06:52] <wikibugs>	 10Fundraising Sprint Potato, 10Fundraising-Backlog, 10fr-donorservices: Fix fake PayPal cancellations on the fly - https://phabricator.wikimedia.org/T242158 (10Ejegg) 05Open→03Resolved
[16:12:13] <wikibugs>	 (03CR) 10Ejegg: "looks pretty conflicted! Is this patch still useful?" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/830956 (https://phabricator.wikimedia.org/T316059) (owner: 10Eileen)
[16:25:52] <wikibugs>	 (03PS1) 10Ejegg: Fix cancel_date for cancelInactives [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/831924 (https://phabricator.wikimedia.org/T315102)
[16:26:15] <ejegg>	 fr-tech anyone got a sec to CR a teeny tiny fix? ^^^
[16:28:50] <wikibugs>	 (03CR) 10Wfan: [C: 03+2] "Looks good, as the date format consistent with others" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/831924 (https://phabricator.wikimedia.org/T315102) (owner: 10Ejegg)
[16:31:45] <ejegg>	 thanks wfan
[16:32:00] <wikibugs>	 (03CR) 10Wfan: [V: 03+2 C: 03+2] Fix cancel_date for cancelInactives [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/831924 (https://phabricator.wikimedia.org/T315102) (owner: 10Ejegg)
[16:42:13] <icinga-wm>	 PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql
[16:42:42] <ejegg>	 wfan is this one OK to deploy? Add last modified date for last delete_deleted_modified_date to skip the check of those contact not able to get deleted f7b3dc717 A bette
[16:43:14] <ejegg>	 https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/830946
[16:43:17] <wfan>	 Yes it's ready to deploy
[16:43:27] <ejegg>	 ok, I'll push it out
[16:43:33] <wfan>	 Thanks elliott :)
[16:43:53] <wikibugs>	 (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/831928
[16:43:59] <wikibugs>	 (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/831928 (owner: 10Ejegg)
[16:47:13] <icinga-wm>	 PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql
[16:49:18] <cstone>	 thanks to whoever was doing the adyen test donations today its helpful data I needed to test the audit
[16:49:49] <wfan>	 np I also needs to do lots adyen hahaha
[16:52:11] <icinga-wm>	 PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql
[16:56:43] <ejegg>	 !log updated fundraising CiviCRM from efbbcb57 to d91b4a2c
[16:56:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:57:13] <icinga-wm>	 PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql
[17:02:13] <icinga-wm>	 PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql
[17:07:13] <icinga-wm>	 PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql
[17:12:13] <icinga-wm>	 PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql
[17:17:13] <icinga-wm>	 PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql
[17:18:29] <dwisehaupt>	 ^^ fixing that up now
[17:37:13] <icinga-wm>	 PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql
[17:38:59] <ejegg|food>	 dwisehaupt: ack, I didn't leave a transaction open when I went for lunch, did I ?
[17:42:13] <icinga-wm>	 PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql
[17:46:50] <dwisehaupt>	 nope. this has been happening with some analytics queries.
[17:46:55] <ejegg>	 ah ok
[17:47:11] <icinga-wm>	 PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql
[17:47:19] <dwisehaupt>	 for some reason certain queries aren't executing correctly over the replication stream but do directly on the host.
[17:47:28] <dwisehaupt>	 mostly around create or replace table commands.
[17:57:13] <icinga-wm>	 RECOVERY - check_mysql on frdb2003 is OK: Uptime: 1819049 Threads: 5 Questions: 939937329 Slow queries: 669 Opens: 4318 Open tables: 1444 Queries per second avg: 516.719 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql
[18:34:02] <wikibugs>	 (03PS7) 10Cstone: WIP: Adding PaymentAccountingReport to the files the adyen audit looks at [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/831159 (https://phabricator.wikimedia.org/T314753)
[18:34:29] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] WIP: Adding PaymentAccountingReport to the files the adyen audit looks at [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/831159 (https://phabricator.wikimedia.org/T314753) (owner: 10Cstone)
[18:37:44] <wikibugs>	 (03PS8) 10Cstone: WIP: Adding PaymentAccountingReport to the files the adyen audit looks at [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/831159 (https://phabricator.wikimedia.org/T314753)
[19:06:57] <wikibugs>	 10Fundraising Sprint Potato, 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Tech-Analytics: Null banner entries for "_ideal" banners/donations - https://phabricator.wikimedia.org/T316949 (10Ejegg) 05Open→03Resolved
[19:07:15] <wikibugs>	 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising-Backlog, 10FR-Braintree-Integration: Paypal audit parser should drop Braintree-initiated donations - https://phabricator.wikimedia.org/T315258 (10Ejegg) 05Open→03Resolved
[19:07:53] <wikibugs>	 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Patch-For-Review: Stop prolific doctor breeding - https://phabricator.wikimedia.org/T315509 (10Ejegg) 05Open→03Resolved
[19:07:58] <wikibugs>	 10Fundraising Sprint NaN is a Number, 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising-Backlog, and 2 others: Investigate queue consumer slowdown in pretest of July 13 2022 - https://phabricator.wikimedia.org/T313000 (10Ejegg)
[19:08:45] <wikibugs>	 10Fundraising Sprint Potato, 10Fundraising-Backlog: Update payments-wiki docker image in fundraising-dev from buster to bullseye - https://phabricator.wikimedia.org/T316848 (10Ejegg) 05Open→03Resolved
[19:08:47] <wikibugs>	 10Fundraising-Backlog, 10Epic: Upgrading payments-wiki from buster to bullseye - https://phabricator.wikimedia.org/T315585 (10Ejegg)
[19:14:00] <wikibugs>	 10Fundraising Sprint Potato, 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Tech-Analytics: Null banner entries for "_ideal" banners/donations - https://phabricator.wikimedia.org/T316949 (10Ejegg)
[19:14:23] <wikibugs>	 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising-Backlog, 10Patch-For-Review: Allow PayPal donations from CN - https://phabricator.wikimedia.org/T316703 (10Ejegg)
[19:14:48] <wikibugs>	 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising-Backlog, 10FR-Braintree-Integration: Paypal audit parser should drop Braintree-initiated donations - https://phabricator.wikimedia.org/T315258 (10Ejegg)
[19:15:29] <wikibugs>	 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Patch-For-Review: Stop prolific doctor breeding - https://phabricator.wikimedia.org/T315509 (10Ejegg)
[19:16:04] <wikibugs>	 10Fundraising Sprint Potato, 10Fundraising-Backlog: Update payments-wiki docker image in fundraising-dev from buster to bullseye - https://phabricator.wikimedia.org/T316848 (10Ejegg)
[19:16:43] <wikibugs>	 10Fundraising Sprint Potato, 10Fundraising-Backlog, 10fr-donorservices: Fix fake PayPal cancellations on the fly - https://phabricator.wikimedia.org/T242158 (10Ejegg)
[19:19:29] <wikibugs>	 10Fundraising-Backlog, 10FR-PayPal-ExpressCheckout, 10FR-Smashpig: Add Idempotency headers to PayPal EC API calls - https://phabricator.wikimedia.org/T317696 (10Ejegg)
[19:20:06] <wikibugs>	 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising-Backlog, 10FR-Smashpig: Use idempotency headers in payment processing requests - https://phabricator.wikimedia.org/T243967 (10Ejegg) 05Open→03Resolved
[19:21:27] <wikibugs>	 10Fundraising Sprint Potato, 10Fundraising-Backlog, 10MW-1.39-notes (1.39.0-wmf.28; 2022-09-05): Remaining campaign markets need Adyen set as default - https://phabricator.wikimedia.org/T316168 (10Ejegg) 05Open→03Resolved
[19:22:05] <wikibugs>	 10Fundraising Sprint Potato, 10Fundraising-Backlog, 10FR-Adyen: Adyen Checkout: Show fail form rather than exception when rejecting for fraud - https://phabricator.wikimedia.org/T299690 (10Ejegg) 05Open→03Invalid
[19:25:38] <wikibugs>	 10Fundraising Sprint NaN is a Number, 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising-Backlog, and 4 others: Add multiple currency support to Braintree - https://phabricator.wikimedia.org/T312808 (10Ejegg) 05Open→03Resolved
[19:26:23] <wikibugs>	 10Fundraising Sprint Potato, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Patch-For-Review: Delete deleted contacts - lets get a bit more done pre BE - https://phabricator.wikimedia.org/T316059 (10Ejegg) 05Open→03Resolved
[19:30:39] <wikibugs>	 10Fundraising Sprint NaN is a Number, 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising-Backlog, and 2 others: Investigate queue consumer slowdown in pretest of July 13 2022 - https://phabricator.wikimedia.org/T313000 (10Eileenmcnaughton) a:03Eileenmcnaughton
[19:30:46] <wikibugs>	 10Fundraising Sprint NaN is a Number, 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising-Backlog, and 2 others: Investigate queue consumer slowdown in pretest of July 13 2022 - https://phabricator.wikimedia.org/T313000 (10Eileenmcnaughton) 05Open→03Resolved
[19:33:39] <wikibugs>	 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising Sprint Potato, 10Fundraising-Backlog, 10FR-form-chooser-refactor: Document new gateway chooser logic and settings - https://phabricator.wikimedia.org/T315011 (10Ejegg) 05Open→03Resolved
[19:36:55] <wikibugs>	 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising Sprint Machine Unlearning, 10Fundraising Sprint NaN is a Number, 10Fundraising Sprint Overused petting Zoo Memetics, and 4 others: configure endpoint monitoring for Braintree API hosts - https://phabricator.wikimedia.org/T311727 (10Ejegg) Fo...
[19:44:29] <wikibugs>	 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices, 10MW-1.39-notes (1.39.0-wmf.27; 2022-08-29): Donations appearing in Civi but not Adyen - https://phabricator.wikimedia.org/T315487 (10Ejegg) 05Open→03Resolved
[19:44:31] <wikibugs>	 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Pending Transaction Resolver Diagram - https://phabricator.wikimedia.org/T316684 (10Ejegg) 05Open→03Resolved
[19:44:33] <wikibugs>	 10Fundraising Sprint NaN is a Number, 10Fundraising-Backlog, 10Epic: [epic] Adyen - auto settle stopped prior donors (Pending transaction Resolver) - https://phabricator.wikimedia.org/T299692 (10Ejegg)
[19:44:35] <wikibugs>	 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Renew PayPal API certificate by September 25th 2022 - https://phabricator.wikimedia.org/T315931 (10Ejegg) 05Open→03Resolved a:03Ejegg
[19:44:38] <wikibugs>	 10Fundraising Tech - Chaos Crew, 10Patch-For-Review: Ensure only paypal supported countries are processed by Braintree - https://phabricator.wikimedia.org/T315684 (10Ejegg) 05Open→03Resolved a:03Ejegg
[19:44:40] <wikibugs>	 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Adyen duplicate donations after donors see error message - August 2022 - https://phabricator.wikimedia.org/T316157 (10Ejegg) 05Open→03Resolved
[19:44:42] <wikibugs>	 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: QueueWrapper should fall back to damaged/retry message table when redis connection is dead - https://phabricator.wikimedia.org/T314805 (10Ejegg) 05Open→03Resolved a:03Ejegg
[19:44:44] <wikibugs>	 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Change source_type to be more specific - https://phabricator.wikimedia.org/T315390 (10Ejegg) 05Open→03Resolved a:03Ejegg
[19:44:51] <wikibugs>	 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10MW-1.39-notes (1.39.0-wmf.27; 2022-08-29): Update Braintree Order ID after successful payment - https://phabricator.wikimedia.org/T314681 (10Ejegg) 05Open→03Resolved a:03Ejegg
[20:19:57] <wikibugs>	 10Fundraising-Backlog: Exhaustively test in fundraising-dev - https://phabricator.wikimedia.org/T317701 (10Cstone)
[20:20:57] <wikibugs>	 10Fundraising-Backlog: Exhaustively test bullseye upgrade in fundraising-dev - https://phabricator.wikimedia.org/T317701 (10Cstone)
[20:21:26] <wikibugs>	 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Adyen getAccountConfig() call failures - https://phabricator.wikimedia.org/T313441 (10Ejegg)
[20:35:18] <wikibugs>	 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Adyen: Client side error on Safari even with successful donation - https://phabricator.wikimedia.org/T310737 (10Ejegg)
[20:36:15] <wikibugs>	 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Donor directed to "Thank You" page even though payment rejected - https://phabricator.wikimedia.org/T310383 (10Ejegg)
[20:37:54] <wikibugs>	 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Deleted Contact Overwriting Current Contact Record - https://phabricator.wikimedia.org/T311438 (10Ejegg)
[20:38:20] <wikibugs>	 10Fundraising-Backlog: Custom field for Direct Mail Appeal not working on Engage import - https://phabricator.wikimedia.org/T317327 (10MDemosWMF) Hi all, I just did an import with gifts that have this DMA and it worked this time! Not sure what changed, but seems to be working now. @RLewis
[21:08:47] <wfan>	 IT service told me to reach out SRE for phab group permission, should I wait for Dylan or  who should I contact here?
[21:11:03] <jgleeson>	 wfan: checkout this old link, it has a shortcut for creating a task to request access on the phab group https://office.wikimedia.org/wiki/Guide_for_new_engineering_staff#Phabricator
[21:11:33] <wfan>	 thanks jgleeson: will do
[21:12:30] <jgleeson>	 catch you all tomorrow o/
[21:16:49] <wfan>	 Yeah ticket created and good night jack
[22:36:14] <wikibugs>	 10Fundraising-Backlog, 10FR-Adyen, 10MediaWiki-extensions-DonationInterface: Stop Adyen adapter from sending an additional pending message on success - https://phabricator.wikimedia.org/T317713 (10Ejegg)
[22:58:34] <dwisehaupt>	 hmmm... donation processing time is a bit jumpy in today's test.
[23:10:41] <ejegg>	 eileen: oh hey, did you turn greeting processing off at some point?
[23:10:53] <eileen>	 no I don't think so
[23:11:00] <eileen>	 oh I meant to didn't I
[23:11:25] <eileen>	 I saw some jumpyness & wondered if it was short runs with high overhead
[23:11:58] <ejegg>	 oh man, that's pretty bad... the create contact line spends a lot of time up near 2 sec per contact
[23:44:47] <dwisehaupt>	 yeah, you can definitely see it in the queue level also: https://frmon.wikimedia.org/d/5OFJxLk4z/dallas-donation-queue?orgId=1&from=1663099408143&to=1663112666075
[23:45:01] <dwisehaupt>	 green is the current test, yellow was the most recent
[23:50:37] <dwisehaupt>	 looks like there are 2 user triggered dedupe jobs running right now. one for 16min one for 20 mins.
[23:51:31] <dwisehaupt>	 tempted to kill those off.
[23:51:45] <dwisehaupt>	 since we are seeing some failmail around db locks.
[23:52:33] <dwisehaupt>	 eileen: any thoughts on killing those off?
[23:53:05] <eileen>	 dwisehaupt: are they def user triggered?
[23:53:11] <eileen>	 not ours?
[23:53:37] <dwisehaupt>	 definitely. /* User : 498 */INSERT INTO civicrm_tmp_e_dedupe_81a06
[23:53:43] <eileen>	 ah
[23:53:53] <eileen>	 well if they are at that insert point def safe to kill
[23:54:10] <dwisehaupt>	 yeah, they are both at that point.
[23:54:18] <eileen>	 probably too big a query
[23:55:16] <dwisehaupt>	 yeah. there were a _lot_ of records on those.
[23:56:21] <dwisehaupt>	 lets see if that helps our queue. but given that they only started 20+mins ago, that wouldn't explain the spikiness. i'll look and see if there is anything in the slow log.