[00:28:15] <wikibugs>	 (03PS5) 10Eileen: Add audit action and allow it to pass out to the settle action. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174561
[00:44:46] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add audit action and allow it to pass out to the settle action. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174561 (owner: 10Eileen)
[01:31:36] <wikibugs>	 (03PS1) 10Ejegg: Ignore Gravy lines in DLocal audit [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178124 (https://phabricator.wikimedia.org/T401689)
[01:31:56] <ejegg>	 eileen: would you mind taking a quick look at that? ^^^
[01:32:02] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Ignore Gravy lines in DLocal audit [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178124 (https://phabricator.wikimedia.org/T401689) (owner: 10Ejegg)
[01:32:12] <ejegg>	 oops, once I get it passing, that is
[01:33:24] <ejegg>	 it's doing the same thing as this did for Adyen: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/SmashPig/+/1070081
[01:34:30] <ejegg>	 ah dang, maybe refunds & chargebacks don't follow the invoice pattern?
[01:42:38] <wikibugs>	 (03PS2) 10Ejegg: Ignore Gravy lines in DLocal audit [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178124 (https://phabricator.wikimedia.org/T401689)
[01:48:17] <wikibugs>	 (03PS1) 10Ejegg: Add types to Dlocal audit functions [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178128
[01:48:27] <ejegg>	 ok, those should be passing eileen ^^
[01:50:51] <ejegg>	 lemme take another look at that settle patch
[01:51:01] <wikibugs>	 (03CR) 10Eileen: [C:03+2] Ignore Gravy lines in DLocal audit [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178124 (https://phabricator.wikimedia.org/T401689) (owner: 10Ejegg)
[01:51:31] <wikibugs>	 (03Merged) 10jenkins-bot: Ignore Gravy lines in DLocal audit [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178124 (https://phabricator.wikimedia.org/T401689) (owner: 10Ejegg)
[01:51:37] <ejegg>	 thanks!
[01:51:56] <wikibugs>	 (03CR) 10Eileen: [C:03+2] Add types to Dlocal audit functions [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178128 (owner: 10Ejegg)
[01:52:25] <wikibugs>	 (03Merged) 10jenkins-bot: Add types to Dlocal audit functions [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178128 (owner: 10Ejegg)
[01:52:58] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] First cut at WMFAudit::settle api (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170226 (https://phabricator.wikimedia.org/T396071) (owner: 10Eileen)
[01:58:10] <wikibugs>	 (03PS1) 10Eileen: Add settled_currency, batch_number, exchange_rate [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178130
[01:58:28] <eileen>	 ejegg: not sure how right ^^ is but
[01:58:35] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add settled_currency, batch_number, exchange_rate [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178130 (owner: 10Eileen)
[01:58:44] <eileen>	 not very :-)
[01:59:55] <wikibugs>	 (03CR) 10Eileen: First cut at WMFAudit::settle api (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170226 (https://phabricator.wikimedia.org/T396071) (owner: 10Eileen)
[02:00:08] <ejegg>	 wait, there's already settlementCurrency in this other patch, right?
[02:00:20] <ejegg>	 https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1178086
[02:00:46] <ejegg>	 oh i see it's using currency ?? USD
[02:04:23] <ejegg>	 going to tag a new SmashPig version and get it into the crm/vendor for the audit
[02:05:17] <eileen>	 cool cool
[02:06:13] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Further adds to the settle api [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178086 (owner: 10Eileen)
[02:08:44] <wikibugs>	 (03PS1) 10Ejegg: Update SmashPig to 0.9.7.7 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178132
[02:09:44] <wikibugs>	 (03PS1) 10Ejegg: Update SmashPig to 0.9.7.7 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1178133
[02:09:48] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Update SmashPig to 0.9.7.7 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1178133 (owner: 10Ejegg)
[02:10:27] <wikibugs>	 (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1178134
[02:10:31] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Update SmashPig to 0.9.7.7 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178132 (owner: 10Ejegg)
[02:10:33] <wikibugs>	 (03Merged) 10jenkins-bot: First cut at WMFAudit::settle api [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170226 (https://phabricator.wikimedia.org/T396071) (owner: 10Eileen)
[02:10:40] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1178134 (owner: 10Ejegg)
[02:20:34] <wikibugs>	 (03Merged) 10jenkins-bot: Further adds to the settle api [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178086 (owner: 10Eileen)
[02:26:10] <wikibugs>	 (03Merged) 10jenkins-bot: Update SmashPig to 0.9.7.7 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1178133 (owner: 10Ejegg)
[02:26:10] <wikibugs>	 (03Merged) 10jenkins-bot: Update SmashPig to 0.9.7.7 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178132 (owner: 10Ejegg)
[02:26:11] <wikibugs>	 (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1178134 (owner: 10Ejegg)
[02:28:12] <ejegg>	 !log fundraising civicrm upgraded from 13fb0ba8 to 321a17c0
[02:28:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[02:36:44] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: dLocal audit timeout failmail - https://phabricator.wikimedia.org/T401689#11080936 (10Ejegg) OK, I deployed a fix for this.
[02:49:13] <wikibugs>	 (03PS6) 10Eileen: Add audit action and allow it to pass out to the settle action. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174561
[02:49:13] <wikibugs>	 (03PS1) 10Eileen: Label test data array [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178139
[02:54:59] <wikibugs>	 (03PS7) 10Eileen: Add audit action and allow it to pass out to the settle action. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174561
[04:51:07] <wikibugs>	 10fundraising-tech-ops, 10Wikimedia-Fundraising-CiviCRM: Can't connect to mysql on staging... - https://phabricator.wikimedia.org/T401759#11081051 (10Jgreen) 05Open→03Resolved p:05Triage→03Unbreak! Fixed. It was a mistake I made when adjusting firewall policy for the new analytics db.
[07:34:43] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178429 (owner: 10L10n-bot)
[07:56:14] <wikibugs>	 (03CR) 10Abijeet Patro: "Looks good to be. Feel free to merge." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178429 (owner: 10L10n-bot)
[11:39:03] <wikibugs>	 06Fundraising-Backlog: Too many requests 529 error - NL - https://phabricator.wikimedia.org/T401821 (10RKumar_WMF) 03NEW
[12:57:40] <wikibugs>	 06Fundraising-Backlog: Too many requests 529 error - NL - https://phabricator.wikimedia.org/T401821#11082443 (10jgleeson) I've merged this into the earlier ticket. The reason for the issue is the same, but I'll update the info about the new error message
[12:59:11] <wikibugs>	 06Fundraising-Backlog: Too many requests 529 error - NL - https://phabricator.wikimedia.org/T401821#11082447 (10Jgreen) From the logs, the IP in question is in the Cloudflare subnet that we think includes their WARP proxy service. The IP got blocked after tripping IPVelocityFilter. ` Aug 13 09:34:30 payments1007...
[12:59:56] <jgleeson>	 more cloudflare fun Jeff_Green 
[13:00:08] <Jeff_Green>	 yeah, I was just looking at that
[13:00:14] <Jeff_Green>	 oh as you see ha ^^^
[13:00:50] <jgleeson>	 :)
[13:01:59] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Too many requests 529 error - NL - https://phabricator.wikimedia.org/T401821#11082470 (10jgleeson)
[13:02:23] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Too many requests 529 error - NL - https://phabricator.wikimedia.org/T401821#11082473 (10jgleeson) p:05Triage→03High a:03jgleeson
[13:02:33] <jgleeson>	 I've moved that to high due to the active campaign 
[13:03:28] <Jeff_Green>	 I'm asking in the slack thread to ask donors if they're using a vpn/proxy
[13:04:01] <Jeff_Green>	 depending on why they're getting routed/aggregated through that subnet, I guess we have a decision to make about how to handle it
[13:07:15] <Jeff_Green>	 jgleeson: is there any reason to investigate further i.e. about transactions leading up to the rate limit?
[13:08:20] <jgleeson>	 not that I can think of Jeff_Green 
[13:09:34] <Jeff_Green>	 ok
[13:10:29] <Jeff_Green>	 any idea why they're still tripping ipvelocityfilter? I thought we had allow-listed the subnet
[13:20:17] <wikibugs>	 (03PS6) 10Damilare Adedoyin: WIP: Create donor portal page for recurring donation cancel process. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178053 (https://phabricator.wikimedia.org/T388749)
[13:22:44] <wikibugs>	 (03PS13) 10Damilare Adedoyin: Create donor portal page for recurring donation pause process. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1176526 (https://phabricator.wikimedia.org/T388749)
[13:25:25] <wikibugs>	 (03CR) 10CI reject: [V:04-1] WIP: Create donor portal page for recurring donation cancel process. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178053 (https://phabricator.wikimedia.org/T388749) (owner: 10Damilare Adedoyin)
[13:26:29] <wikibugs>	 (03PS7) 10Damilare Adedoyin: WIP: Create donor portal page for recurring donation cancel process. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178053 (https://phabricator.wikimedia.org/T388749)
[13:30:03] <wikibugs>	 (03CR) 10CI reject: [V:04-1] WIP: Create donor portal page for recurring donation cancel process. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178053 (https://phabricator.wikimedia.org/T388749) (owner: 10Damilare Adedoyin)
[13:50:00] <wikibugs>	 (03PS8) 10Damilare Adedoyin: WIP: Create donor portal page for recurring donation cancel process. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178053 (https://phabricator.wikimedia.org/T388749)
[13:51:20] <wikibugs>	 (03PS9) 10Damilare Adedoyin: WIP: Create donor portal page for recurring donation cancel process. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178053 (https://phabricator.wikimedia.org/T388749)
[13:52:18] <jgleeson>	 good point Jeff_Green. I'll see if that allow-list actually impacts the bans. It might just be allowing the header swap out
[13:53:12] <Jeff_Green>	 I'm testing WARP with a VM, so far it looks like testing is working and my web requests route through that subnet
[13:54:14] <Jeff_Green>	 and there are no headers added, interestingly I can pass through XFF and CF-Connecting-IP headers so they aren't removing headers either
[13:59:07] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Too many requests 529 error - NL - https://phabricator.wikimedia.org/T401821#11082709 (10MBeat33)
[14:05:51] <wikibugs>	 (03PS10) 10Damilare Adedoyin: WIP: Create donor portal page for recurring donation cancel process. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178053 (https://phabricator.wikimedia.org/T388749)
[14:06:18] <wikibugs>	 (03PS11) 10Damilare Adedoyin: WIP: Create donor portal page for recurring donation cancel process. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178053 (https://phabricator.wikimedia.org/T388749)
[14:33:26] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Too many requests 529 error - NL - https://phabricator.wikimedia.org/T401821#11082836 (10Jgreen) Correction re. logs, it was ~not~ the IPVelocityFilter messages that triggered fail2ban, it was "Redirecting for transaction"
[14:42:10] <wikibugs>	 06Fundraising-Backlog, 10FR-Smashpig, 10Recurring-Donations: Send failmail when recurring charge API call gets a validation error from Gravy - https://phabricator.wikimedia.org/T401834 (10Ejegg) 03NEW
[14:43:29] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Civi: multiple issues with recurring donations - https://phabricator.wikimedia.org/T399339#11082865 (10Ejegg) Added a new task to start sending failmail on validation errors: T401834
[14:46:22] <wikibugs>	 10Fundraising analytics stack: fix dagster jobs relying on gsheets dlt script - https://phabricator.wikimedia.org/T401835 (10AStein-WMF) 03NEW
[14:47:50] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Civi: multiple issues with recurring donations - https://phabricator.wikimedia.org/T399339#11082891 (10MSuijkerbuijk_WMF) Thanks
[15:06:57] <wikibugs>	 06Fundraising-Backlog: Upgrade fr-tech proxy VM to Debian trixie - https://phabricator.wikimedia.org/T401838 (10Ejegg) 03NEW
[15:12:44] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: URGENT: Merge Behavior Changed — Causing RML Emails to Be Sent in Error! - https://phabricator.wikimedia.org/T401752#11082976 (10jgleeson)
[15:25:00] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: URGENT: Merge Behavior Changed — Causing RML Emails to Be Sent in Error! - https://phabricator.wikimedia.org/T401752#11083017 (10greg) p:05Triage→03Unbreak!
[15:29:24] <wikibugs>	 (03PS12) 10Damilare Adedoyin: WIP: Create donor portal page for recurring donation cancel process. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178053 (https://phabricator.wikimedia.org/T388749)
[15:32:15] <wikibugs>	 (03CR) 10CI reject: [V:04-1] WIP: Create donor portal page for recurring donation cancel process. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178053 (https://phabricator.wikimedia.org/T388749) (owner: 10Damilare Adedoyin)
[15:41:41] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: URGENT: Merge Behavior Changed — Causing RML Emails to Be Sent in Error! - https://phabricator.wikimedia.org/T401752#11083098 (10jgleeson) a:03jgleeson
[15:48:12] <icinga-wm>	 PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5576 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql
[15:52:41] <wikibugs>	 10fundraising-tech-ops: dns_to_ipset should clean up deprecated tables - https://phabricator.wikimedia.org/T401419#11083127 (10Jgreen) 05Open→03Resolved a:03Jgreen This is done!
[15:53:14] <icinga-wm>	 PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4092 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql
[15:56:10] <icinga-wm>	 PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22082 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql
[15:56:14] <icinga-wm>	 PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22087 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql
[15:58:16] <icinga-wm>	 PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1543 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql
[15:59:08] <icinga-wm>	 ACKNOWLEDGEMENT - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1543 Jeff_Green Known. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql
[15:59:08] <icinga-wm>	 ACKNOWLEDGEMENT - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22082 Jeff_Green Known. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql
[15:59:09] <icinga-wm>	 ACKNOWLEDGEMENT - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22087 Jeff_Green Known. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql
[16:03:14] <icinga-wm>	 RECOVERY - check_mysql on frdb1004 is OK: Uptime: 1025751 Threads: 4 Questions: 80069575 Slow queries: 525 Opens: 1624 Open tables: 1330 Queries per second avg: 78.059 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql
[16:12:48] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: URGENT: Merge Behavior Changed — Causing RML Emails to Be Sent in Error! - https://phabricator.wikimedia.org/T401752#11083229 (10jgleeson) Using this ticket as a notepad.  I did some digging and found that the secondary email exists...
[16:18:37] <wikibugs>	 (03CR) 10Wfan: [C:03+2] Set country custom field on contribution_recur [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177454 (https://phabricator.wikimedia.org/T392376) (owner: 10Ejegg)
[16:18:56] <wikibugs>	 (03CR) 10Wfan: [C:03+2] Use original_country in recurring charge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177474 (https://phabricator.wikimedia.org/T392376) (owner: 10Ejegg)
[16:38:58] <wikibugs>	 (03Merged) 10jenkins-bot: Set country custom field on contribution_recur [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177454 (https://phabricator.wikimedia.org/T392376) (owner: 10Ejegg)
[16:39:06] <wikibugs>	 (03Merged) 10jenkins-bot: Use original_country in recurring charge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1177474 (https://phabricator.wikimedia.org/T392376) (owner: 10Ejegg)
[16:40:14] <wikibugs>	 10fundraising-tech-ops: Fundraising systems Debian Trixie support - https://phabricator.wikimedia.org/T401847 (10Jgreen) 03NEW
[16:56:08] <icinga-wm>	 RECOVERY - check_mysql on frdb2004 is OK: Uptime: 174285 Threads: 4 Questions: 17176154 Slow queries: 91 Opens: 1301 Open tables: 1223 Queries per second avg: 98.552 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15819 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql
[17:06:08] <icinga-wm>	 PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12334 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql
[17:11:10] <icinga-wm>	 PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10658 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql
[17:15:50] <wikibugs>	 (03CR) 10Ejegg: "OK, we can try this again. Gravy has just deployed a change to skip sending incomplete buyer information to Trustly." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1176301 (https://phabricator.wikimedia.org/T401277) (owner: 10Ejegg)
[17:16:27] <wikibugs>	 (03PS5) 10Ejegg: Restore buyer email param on ACH request [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1176301 (https://phabricator.wikimedia.org/T401277)
[17:17:02] <icinga-wm>	 PROBLEM - Host frbast1002 is DOWN: PING CRITICAL - Packet loss = 100%
[17:22:04] <icinga-wm>	 RECOVERY - Host frbast1002 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms
[17:38:22] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 07payments-orchestration: Apple Pay donation assigned to wrong contact in CiviCRM despite correct name in Gravy and Adyen - https://phabricator.wikimedia.org/T396026#11083492 (10jgleeson)
[17:40:03] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 07payments-orchestration: Source of Truth Unclear: Boleto Donation Mismatch Between Gravy, dLocal, and Civi - https://phabricator.wikimedia.org/T400479#11083494 (10jgleeson)
[17:41:08] <icinga-wm>	 RECOVERY - check_mysql on frdb2004 is OK: Uptime: 176985 Threads: 4 Questions: 22764707 Slow queries: 93 Opens: 1320 Open tables: 1225 Queries per second avg: 128.625 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql
[17:42:58] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 07payments-orchestration: Two recurring donations missing from CiviCRM. - https://phabricator.wikimedia.org/T399471#11083502 (10jgleeson)
[17:43:54] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 07payments-orchestration: donor name transposed from Gravy to Civi: issue? - https://phabricator.wikimedia.org/T397860#11083504 (10jgleeson)
[17:44:02] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 07payments-orchestration: Settled Adyen Donation - Settled Gravy Donation - Not Appearing in Civi - https://phabricator.wikimedia.org/T395340#11083505 (10jgleeson)
[17:44:19] <wikibugs>	 06Fundraising-Backlog, 07payments-orchestration: Helping Gr4vy with Venmo setup - https://phabricator.wikimedia.org/T400413#11083507 (10Damilare) Gravy requires a test in production to finalise the work on this. According to the [payment routing doc,](https://docs.google.com/spreadsheets/d/1Qe3PAcl2P3qsXGxIMWo...
[17:44:35] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 07payments-orchestration: Pending transaction resolver - two transactions resolved in the same day - https://phabricator.wikimedia.org/T394103#11083508 (10jgleeson)
[17:45:09] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 07payments-orchestration: Refund shows as contribution line without original contribution line - https://phabricator.wikimedia.org/T393036#11083512 (10jgleeson)
[17:46:49] <wikibugs>	 06Fundraising-Backlog, 07payments-orchestration: Helping Gr4vy with Venmo setup - https://phabricator.wikimedia.org/T400413#11083517 (10Damilare)
[17:46:50] <wikibugs>	 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 07Epic, 07payments-orchestration: Gravy Braintree implementation: Venmo - https://phabricator.wikimedia.org/T375267#11083516 (10Damilare)
[17:51:14] <icinga-wm>	 RECOVERY - check_mysql on frdb2005 is OK: Uptime: 1032647 Threads: 4 Questions: 81618030 Slow queries: 551 Opens: 1586 Open tables: 1286 Queries per second avg: 79.037 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql
[17:53:32] <wikibugs>	 (03CR) 10Jgleeson: [C:03+2] "this worked for me when testing https://sandbox.wikimedia.gr4vy.app/merchants/default/transactions/3868bcae-f01e-49ec-a4da-7a753fe8d8db/ov" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1176301 (https://phabricator.wikimedia.org/T401277) (owner: 10Ejegg)
[17:54:27] <wikibugs>	 (03Merged) 10jenkins-bot: Restore buyer email param on ACH request [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1176301 (https://phabricator.wikimedia.org/T401277) (owner: 10Ejegg)
[17:55:13] <wikibugs>	 (03CR) 10Ejegg: "Seems to work in smoke tests so far." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1176301 (https://phabricator.wikimedia.org/T401277) (owner: 10Ejegg)
[17:56:56] <ejegg>	 thanks jgleeson 
[17:57:02] <jgleeson>	 np!
[18:06:36] <wikibugs>	 (03PS2) 10Eileen: Label test data array [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178139
[18:06:40] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Label test data array [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178139 (owner: 10Eileen)
[18:22:39] <wikibugs>	 (03Merged) 10jenkins-bot: Label test data array [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178139 (owner: 10Eileen)
[18:37:56] <wikibugs>	 06Fundraising-Backlog: RML Country information not making it into Civi from Acoustic - https://phabricator.wikimedia.org/T400786#11083697 (10SHust) @Eileenmcnaughton turns out this is still broken!  Example: Ticket #1710951 / CID 68135552, where the country doesn’t show up on the Civi summary page for the RML re...
[18:38:12] <wikibugs>	 06Fundraising-Backlog, 06FR-donorrelations: RML Country information not making it into Civi from Acoustic - https://phabricator.wikimedia.org/T400786#11083698 (10SHust)
[19:44:05] <wikibugs>	 06Fundraising-Backlog, 06FR-donorrelations: Gravy-PayPal duplicate donation in Civi: what to do - https://phabricator.wikimedia.org/T401866 (10MBeat33) 03NEW
[19:49:45] <wikibugs>	 06Fundraising-Backlog, 06FR-donorrelations: Gravy-PayPal duplicate donation in Civi: what to do - https://phabricator.wikimedia.org/T401866#11083948 (10MBeat33)
[19:54:00] <icinga-wm>	 PROBLEM - check_load on fransw2003 is CRITICAL: LOAD CRITICAL - total load average: 128.64, 73.84, 46.03 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_load
[19:54:02] <icinga-wm>	 PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 100% used. Largest process: trino-server-co (997752) = 99.0% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory
[19:56:14] <icinga-wm>	 RECOVERY - check_memory on fransw2003 is OK: OK Memory 0% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory
[20:01:14] <icinga-wm>	 RECOVERY - check_load on fransw2003 is OK: LOAD OK - total load average: 1.22, 25.74, 38.22 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_load
[20:07:18] <wikibugs>	 (03PS1) 10Damilare Adedoyin: Remove validation for venmo payments [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1178614 (https://phabricator.wikimedia.org/T400413)
[20:09:29] <damilare>	 fr-tech here are the vue guidelines by mediawiki https://www.mediawiki.org/wiki/Vue.js/Guidelines
[20:12:57] <wikibugs>	 (03PS13) 10Damilare Adedoyin: WIP: Create donor portal page for recurring donation cancel process. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178053 (https://phabricator.wikimedia.org/T388749)
[20:20:15] <jgleeson>	 thanks damilare 
[20:20:45] <damilare>	 np!
[20:25:04] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: URGENT: Merge Behavior Changed — Causing RML Emails to Be Sent in Error! - https://phabricator.wikimedia.org/T401752#11084023 (10Eileenmcnaughton) I believe there are 35 affected contacts  SELECT c.id, MAX(c.modified_date) FROM silv...
[21:06:10] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2403 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[21:11:14] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2703 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[21:16:14] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3003 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[21:21:12] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3303 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[21:26:14] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3603 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[21:31:14] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3903 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[21:36:12] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4203 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[21:41:14] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4503 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[21:46:12] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4803 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[21:51:14] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5104 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[21:56:14] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5403 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[22:01:13] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5703 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[22:04:12] <wikibugs>	 (03CR) 10Wfan: [C:03+2] Restore buyer email param on ACH request [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1176301 (https://phabricator.wikimedia.org/T401277) (owner: 10Ejegg)
[22:06:13] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6003 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[22:11:13] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6303 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[22:16:11] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6603 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[22:17:58] <wikibugs>	 06Fundraising-Backlog, 06FR-donorrelations: RML Country information not making it into Civi from Acoustic - https://phabricator.wikimedia.org/T400786#11084261 (10Eileenmcnaughton) @SHust that contact was RMLd on July 30 - do you have any that were added to the RML group after 6 Aug  with no country? https://ci...
[22:21:14] <icinga-wm>	 PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3663 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[22:26:14] <icinga-wm>	 RECOVERY - check_mysql on frdb1006 is OK: Uptime: 16533 Threads: 4 Questions: 1111917 Slow queries: 18763 Opens: 1771 Open tables: 1757 Queries per second avg: 67.254 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql
[23:48:24] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: URGENT: Merge Behavior Changed — Causing RML Emails to Be Sent in Error! - https://phabricator.wikimedia.org/T401752#11084352 (10Eileenmcnaughton) Actually it is less than 25 - the cluster modified date of 13-08-25 were ones that wo...