[01:21:16] (03PS1) 10Eileen: Assume import is an individual when donor advised fund specified [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/743056 (https://phabricator.wikimedia.org/T296861) [01:46:30] 10Fundraising Sprint Visual C Saw, 10Fundraising-Backlog, 10fundraising sprint Universal Cereal Bus, 10fundraising sprint Wireless Zipline, and 3 others: Civi: EOY Auto Thank You Email Receipt - New content is ready for coding - https://phabricator.wikimedia.org/T290253 (10Eileenmcnaughton) @TomaszGorski @... [02:02:17] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising sprint Wireless Zipline, 10fr-donorservices: Recurring "Cancel" button reposition request - https://phabricator.wikimedia.org/T291133 (10Cstone) a:03Cstone [02:31:50] 10Fundraising-Backlog, 10fr-donorservices: Amazon donations TY email doesn’t have the donor's name - https://phabricator.wikimedia.org/T296881 (10KHancock99) Also, just to piggyback here ... **Apply Pay** donations are coming up as: - 'Dad' as the donor's full name in Civi - 'Dear donor' in the TY recei... [03:03:43] 10Fundraising-Backlog, 10fr-donorservices: Amazon donations TY email doesn’t have the donor's name - https://phabricator.wikimedia.org/T296881 (10KHancock99) In addition to 'Dad', **Apple Pay** donors are also having their name registered as 'Work' in Civi, and as 'Work' in the subject line of the TY email.... [03:10:25] 10Fundraising-Backlog, 10fr-donorservices: Amazon donations TY email doesn’t have the donor's name - https://phabricator.wikimedia.org/T296881 (10Cstone) Since you can have multiple cards I'm guessing they have them tagged (using the name field) as the family members. (I can see myself having a Parents one if... [03:55:50] 10Fundraising-Backlog, 10fundraising sprint Wireless Zipline, 10fr-donorservices: Civi is no longer displaying the recurring amount in the same format - correctly. - https://phabricator.wikimedia.org/T296663 (10Ejegg) Looks like we have two different code paths that can create recurring records for PayPal.... [03:56:20] haha ejegg I was just looking at that too [03:57:40] explains why some are CAD/CAD and some USD/CAD [04:10:57] 10Fundraising-Backlog, 10fr-donorservices: Amazon donations TY email doesn’t have the donor's name - https://phabricator.wikimedia.org/T296881 (10Ejegg) So we logged the raw response from Amazon for the latest donation of donor 1282579, and the [Name] field they sent to us really did only have the donor's firs... [04:12:12] cstone oops, i shoulda claimed it in phab first... [04:12:32] well, I'm not going to do any more investigations tonight... have at 'em [04:13:35] oh no worries I just looked a little in the db [04:13:52] I didn't claim it either :P [07:45:11] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1256 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [07:47:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1312 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [07:50:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1472 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [07:51:37] jgleeson|away, ejegg|away, AndyRussG, cstone: lag again and repeating alerts ^ [07:52:01] * RhinosF1 doesn't not understand why a dev server needs to send critical alerts every 5 minutes when most of you are asleep [07:52:13] s/not/ [08:32:16] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 1420138 Threads: 18 Questions: 154612331 Slow queries: 978 Opens: 613192805 Flush tables: 1 Open tables: 303 Queries per second avg: 108.871 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2058 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [09:55:16] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 1424693 Threads: 15 Questions: 128845387 Slow queries: 1165668 Opens: 664446991 Flush tables: 1 Open tables: 292 Queries per second avg: 90.437 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 117 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [13:41:03] finally! [13:41:18] fr-tech anyone else had any issues getting on irc today? [13:45:39] looks like we're gonna have a bunch of new failmails to talk about in today's alerts retro. [14:49:26] 10Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for Emma Hughes - https://phabricator.wikimedia.org/T296940 (10spatton) [14:50:25] 10Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for Emma Hughes - https://phabricator.wikimedia.org/T296940 (10spatton) [14:52:10] PROBLEM - check_load on frdb1003 is CRITICAL: CRITICAL - load average: 42.72, 31.82, 20.51 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_load [14:57:10] PROBLEM - check_load on frdb1003 is CRITICAL: CRITICAL - load average: 56.04, 44.56, 28.92 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_load [15:02:10] PROBLEM - check_load on frdb1003 is CRITICAL: CRITICAL - load average: 55.31, 51.39, 36.19 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_load [15:07:10] PROBLEM - check_load on frdb1003 is CRITICAL: CRITICAL - load average: 4.47, 24.73, 29.44 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_load [15:12:10] PROBLEM - check_load on frdb1003 is CRITICAL: CRITICAL - load average: 9.24, 15.23, 24.00 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_load [15:17:10] PROBLEM - check_load on frdb1003 is CRITICAL: CRITICAL - load average: 12.45, 12.73, 20.41 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_load [15:23:44] fr-tech: I've got a conflict with standup this morning. Just my usual update of a bunch of meetings. [15:24:37] Hey XenoRyet no worries of course :) [15:52:11] RECOVERY - check_load on frdb1003 is OK: OK - load average: 0.61, 0.93, 4.19 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_load [16:17:59] fr-tech i'm looking at the latest forgetme failmail [16:18:37] oh wait, the jobs currently in there are not old enough to trigger it [16:18:49] someone must have gotten the 3 hr ago one already [16:20:20] ejegg: any thoughts on how we can fix the thank you mail rep lag pitfall? [16:20:32] could we give that check a retry or two? [16:20:52] we had like 80+ failmails today when I looked [16:22:56] jgleeson: we could force it to use the primary db rather than a replica [16:23:15] i think if we just went back to the drupal query syntax that wouldn't be intercepted by RPOW [16:23:48] ah yeah [16:36:50] paste them together.. [16:37:10] c-suite centipede? [16:37:25] sorry [16:43:35] :) [17:23:00] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Thank you fail mail - https://phabricator.wikimedia.org/T296813 (10Ejegg) @Eileenmcnaughton suggested pointing the query at the primary DB rather than the replica - Would an easy way of doing be to go back to the drupal db_query syntax so it's not interce... [17:29:20] 10Fundraising-Backlog: Database Cleanup: Paypal recurrings with amount of in contribution_recur - https://phabricator.wikimedia.org/T296954 (10Cstone) [17:29:26] as of ~ 1900UTC yesterday, the db read handle is pointed back at the primary db. so if we are still seeing failmails, they shouldn't ve related to db lag. [17:29:44] dwisehaupt: oh interesting [17:34:52] 10Fundraising-Backlog, 10fundraising sprint Wireless Zipline, 10fr-donorservices: Amazon donations TY email doesn’t have the donor's name - https://phabricator.wikimedia.org/T296881 (10DStrine) [17:40:58] 10Fundraising-Backlog, 10fundraising sprint Wireless Zipline, 10fr-donorservices: Amazon donations TY email doesn’t have the donor's name - https://phabricator.wikimedia.org/T296881 (10Ejegg) So one solution would be to add name fields to the Amazon and Apple Pay forms. As this would increase friction, we'd... [17:54:20] 10Fundraising-Backlog: Transactions at status 600 - https://phabricator.wikimedia.org/T296956 (10EMartin) [18:00:23] jgleeson: sorry if i was breaking up on the call. i was trying to get across that most production related queries should now be coming off of frdb1003 vs frdev1001 for things like statler and such. [18:00:52] there may be some processes still running on frdev but i'm unsure if they would be affected by any replication lag. [18:01:40] i would like to calm down the warnings/alerts where possible, especially since they can cause an idea of instability even if that isn't the case. [18:07:43] thanks dwisehaupt [18:12:34] 10Fundraising-Backlog: Adyen ApplePay Issuer Unavailable - https://phabricator.wikimedia.org/T296845 (10Ejegg) 'Issuer' usually refers to the bank issuing the credit card, so maybe a problem affecting network traffic to and from Adyen? [18:23:56] ejegg: i was wondering if it was a specific issuer [18:24:22] but looks like there was a few different types e.g. visa, discovery [18:24:38] ahh [18:34:41] (03PS1) 10Ejegg: Run TY initial query via Civi query function [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/743225 (https://phabricator.wikimedia.org/T296813) [19:51:08] 10Fundraising-Backlog, 10fundraising sprint Wireless Zipline, 10fr-donorservices: Amazon donations TY email doesn’t have the donor's name - https://phabricator.wikimedia.org/T296881 (10Cstone) For the TY subject line being different, it's just checking for first name instead of first name and last name like... [20:00:43] 10Fundraising-Backlog, 10fundraising sprint Wireless Zipline, 10fr-donorservices: Amazon and applepay donations TY email doesn’t have the donor's name - https://phabricator.wikimedia.org/T296881 (10DStrine) p:05Triage→03High [20:20:39] (03PS1) 10Ejegg: Pending resover: incorporate scores from fredge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/743252 (https://phabricator.wikimedia.org/T296555) [20:20:54] ok damilare / jgleeson ^^^ is the WIP from the call [20:21:20] let me know if the commit message or TODOs are confusing [20:26:30] ok I can see how that overlaps with the other ticket we added [20:26:49] which answers the question, are they distinct [20:32:41] (03CR) 10jerkins-bot: [V: 04-1] Pending resover: incorporate scores from fredge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/743252 (https://phabricator.wikimedia.org/T296555) (owner: 10Ejegg) [20:39:01] jgleeson: sorry, are what distinct? [20:41:46] the step to incorporate the scores and the antifraud queue message push step [20:42:01] looks like the latter is dependant on the former [20:42:07] jgleeson: I think we would want to do them in one patch, yeah [20:42:32] so we know whether we actually have any new data, or whether the CVV/AVS scores were already in fredge [20:49:30] oh but that could totally be split across two patches, i guess [20:49:51] we would just do the patch to retrieve the scores first [21:29:25] 10Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for Emma Hughes - https://phabricator.wikimedia.org/T296940 (10Dwisehaupt) [21:30:18] 10Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for Emma Hughes - https://phabricator.wikimedia.org/T296940 (10Dwisehaupt) Approval from Lisa: ` From: Lisa Gruwell To: Samuel Patton Cc: Fundraising Tech Ops Subject: Re: Civi and Superset access for Emma Hughes, our Sr. Frontend Devel... [21:58:21] damilare: I'll catch you tomorrow on our call and we can work out if it's possible to split up the antifraud stuff although if you feel like you have a good handle on it I can hold off and maybe finish off the PendingTransactionResolverQueueConsumer bits we we're looking at adding. either way I'm easy! [21:58:27] were* [21:59:03] have a good rest of your day fr-tech [21:59:13] jgleeson|away: thanks same! [22:00:10] fr-tech I have to go do an errand, back in an hour or so [22:04:56] fr-tech: Same for me actually, gotta go pick up a car part, I'll be back in a bit. [22:43:20] PROBLEM - check_puppetrun on payments2001 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[fetch_geoip_database] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_puppetrun [22:45:16] PROBLEM - check_puppetrun on frauth2001 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[fetch_geoip_database] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2001&service=check_puppetrun [22:48:20] RECOVERY - check_puppetrun on payments2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_puppetrun [22:50:16] RECOVERY - check_puppetrun on frauth2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2001&service=check_puppetrun [22:51:20] PROBLEM - check_puppetrun on payments2002 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 5 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[fetch_geoip_database] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_puppetrun [22:53:20] PROBLEM - check_puppetrun on payments2003 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 7 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[fetch_geoip_database] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_puppetrun [22:54:28] these puppet problems are just state issues. they will error and then clear. [22:56:20] RECOVERY - check_puppetrun on payments2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_puppetrun [22:58:20] RECOVERY - check_puppetrun on payments2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_puppetrun [23:57:30] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Paypal_endpoint_critical 3 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [23:59:53] getting a few read timeouts for the paypal endpoint. may be transient.