[01:32:12] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Amazon:1) 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [01:37:12] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [04:28:43] (03PS2) 10Eileen: Move our transaction handling class to extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/922945 [04:28:45] (03PS2) 10Eileen: Minor extraction [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/922947 [04:28:47] (03PS1) 10Eileen: Add conditional to help with tests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/924178 [04:31:45] (03PS1) 10Eileen: Declare field names in one function only [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/924179 [04:42:03] (03CR) 10CI reject: [V: 04-1] Add conditional to help with tests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/924178 (owner: 10Eileen) [04:45:54] (03CR) 10CI reject: [V: 04-1] Declare field names in one function only [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/924179 (owner: 10Eileen) [05:06:36] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_39) - 10https://gerrit.wikimedia.org/r/924219 (owner: 10L10n-bot) [05:25:16] (03CR) 10Abijeet Patro: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_39) - 10https://gerrit.wikimedia.org/r/924219 (owner: 10L10n-bot) [15:34:03] 10fundraising-tech-ops, 10FR-Email: Update Acoustic Unsubscribe landing page domain to be served by a secure wikimedia.org domain - https://phabricator.wikimedia.org/T336000 (10EWilfong_WMF) @Dwisehaupt - I asked about this and Acoustic sent the following response: > The links domain and the custom landing pa... [15:49:19] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Thank you job was disabled!!! - https://phabricator.wikimedia.org/T337281 (10Ejegg) Hi @greg, I still just want to understand why the nagios alert didn't trigger in this case - at least, I don't remember seeing any nagio... [16:28:27] 10Fundraising-Backlog, 10fundraising-tech-ops: migrate Silverpop export job from civi1001/frdev1001 to civi1002+frdb1006 - https://phabricator.wikimedia.org/T337498 (10Ejegg) @jgreen this looks pretty complete to me. Besides the script, we humans check in on the data and occasionally need to manually run updat... [17:08:59] 10Fundraising-Backlog: Migrate phab task description to cowsay format - https://phabricator.wikimedia.org/T337046 (10AKanji-WMF) a:03Cstone {meme, src="goat-for-it"} [17:12:05] LOL ^ [17:12:51] there were a couple of other easter eggs hidden in phab [17:13:02] cats and stuff [17:29:04] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Civi - Increase limit on email sends to 2000 - https://phabricator.wikimedia.org/T335296 (10AKanji-WMF) I'm trying to arrange a call between Rosie & Eileen to discuss alternate sending methods. [18:18:15] 10Fundraising-Backlog, 10FR-dlocal, 10Recurring-Donations, 10fr-donorservices: recurring dLocal “Wellness” check May 19th - https://phabricator.wikimedia.org/T337049 (10AKanji-WMF) After discussing with @EMartin and @MBeat33 sounds like we will wait until next billing cycle. Closing! [18:25:20] 10Fundraising-Backlog, 10Epic: EPIC: Recurring upsell for donors - https://phabricator.wikimedia.org/T143429 (10Dwisehaupt) [18:46:14] PROBLEM - check_memory on fran1001 is CRITICAL: CRIT Memory 96% used. Largest process: python3 (215220) = 54.9% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_memory [18:51:14] PROBLEM - check_memory on fran1001 is CRITICAL: CRIT Memory 95% used. Largest process: python3 (215220) = 54.9% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_memory [18:56:14] RECOVERY - check_memory on fran1001 is OK: OK Memory 51% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_memory [19:09:03] (03PS1) 10Damilare Adedoyin: Update civiproxy mount order [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/924594 [19:41:35] (03PS2) 10Damilare Adedoyin: Update civiproxy mount order [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/924594 [19:57:24] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Thank you job was disabled!!! - https://phabricator.wikimedia.org/T337281 (10greg) >>! In T337281#8889272, @Ejegg wrote: > Hi @greg, I still just want to understand why the nagios alert didn't trigger in this case - at l... [19:59:25] 10Fundraising-Backlog: Nagios alerts for thank-you emails, verify functionality - https://phabricator.wikimedia.org/T337795 (10greg) [19:59:55] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Thank you job was disabled!!! - https://phabricator.wikimedia.org/T337281 (10greg) Follow-up task filed as T337795, moving this one to done. [20:13:39] 10Fundraising-Backlog: Nagios alerts for thank-you emails, verify functionality - https://phabricator.wikimedia.org/T337795 (10Jgreen) According to the logs, missing_thank_yous reported to icinga from that check peaked at 27. The alert thresholds are 1500/warn and 2000/critical. [21:12:10] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [21:17:10] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [21:22:10] RECOVERY - check_mysql on frdb2003 is OK: Uptime: 942889 Threads: 4 Questions: 195813328 Slow queries: 395 Opens: 3422 Open tables: 1833 Queries per second avg: 207.673 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [21:33:53] 10fundraising-tech-ops, 10FR-Email: Update Acoustic Unsubscribe landing page domain to be served by a secure wikimedia.org domain - https://phabricator.wikimedia.org/T336000 (10Dwisehaupt) @greg @RobH Could you give some advice on how we might proceed with this? We (fr-tech) don't manage ssl certs in this mann... [21:39:20] 10Fundraising-Backlog, 10FR-Smashpig, 10FR-dlocal, 10MediaWiki-extensions-DonationInterface: Stop sending failmail for dlocal 'User blacklisted' or 'User limit exceeded' - https://phabricator.wikimedia.org/T337800 (10Ejegg) [21:40:09] 10Fundraising-Backlog, 10FR-Smashpig, 10FR-dlocal, 10MediaWiki-extensions-DonationInterface: Stop sending failmail for dlocal 'User blacklisted' or 'User limit exceeded' - https://phabricator.wikimedia.org/T337800 (10Ejegg) [21:42:52] (03CR) 10Jgleeson: [V: 03+2 C: 03+2] "Thanks for this! I'm slightly confused why it worked when originally set up with this config but at least now we know it definitely works!" [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/924594 (owner: 10Damilare Adedoyin) [21:50:04] 10Fundraising-Backlog: Nagios alerts for thank-you emails, verify functionality - https://phabricator.wikimedia.org/T337795 (10Jgreen) I am stumped. As shown above, the icinga vs prometheus check run the same query on the same database, though on two different machines (frdb1005 vs frdev1001). The queries report... [22:03:55] (03PS1) 10Ejegg: Preview TY in preferred language [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/924601 [22:04:26] (03PS5) 10Ejegg: Fix rendering of contribution custom fields in preview [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/922637 (https://phabricator.wikimedia.org/T337332) (owner: 10Eileen) [22:04:40] (03CR) 10Ejegg: [C: 03+2] "Thanks, looks good!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/922637 (https://phabricator.wikimedia.org/T337332) (owner: 10Eileen) [22:04:45] (03PS2) 10Ejegg: Preview TY in preferred language [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/924601 [22:05:17] eileen: thanks to your nicely organized code, it was pretty trivial to update that preview language ^^ [22:05:50] (03CR) 10Eileen: [C: 03+2] Preview TY in preferred language [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/924601 (owner: 10Ejegg) [22:06:09] oh nice - that was a definite oversight! [22:06:51] with all the code so far out I feel better digging in cos we have mostly the same code paths now I think (& possibly even cstone's pet hate is fixed but I haven't tested that properly yet) [22:07:13] oh, which was cstone's pet peeve? [22:08:14] the 1 hour cache [22:08:22] ah yeah [22:10:42] 10Fundraising-Backlog: Nagios alerts for thank-you emails, verify functionality - https://phabricator.wikimedia.org/T337795 (10Ejegg) Ah @Jgreen I just noticed the time limits on that query are for donations between 1 month and 3 days old. Let's maybe make that between 1 month and 3 hours old instead. I don't un... [22:15:52] wooo eileen even if its just a possibly still exciting [22:17:02] (03Merged) 10jenkins-bot: Fix rendering of contribution custom fields in preview [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/922637 (https://phabricator.wikimedia.org/T337332) (owner: 10Eileen) [22:18:54] (03Merged) 10jenkins-bot: Preview TY in preferred language [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/924601 (owner: 10Ejegg) [22:19:05] I'll deploy those now eileen [22:19:15] cool - thanks [22:19:42] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/924602 [22:20:22] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/924602 (owner: 10Ejegg) [22:21:19] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/924602 (owner: 10Ejegg) [22:22:23] !log civicrm upgraded from 415aa7e5 to 5905a403 [22:22:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:23:07] ok, the preview renders just fine for that fr_FR person with the failmail [22:24:36] oh right, we changed the footer to use the trxn_id instead of the CNTCT- [22:25:16] wording seems a little funny though, since it still refers to 'your donation, number RECURRING ADYEN MSDLKAJSDI123ASD2143' [22:30:59] gonna help with dinner, back later [23:02:48] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops: Do a prune on detail data for activity records of type thank you - https://phabricator.wikimedia.org/T325123 (10Eileenmcnaughton) Note that we now log all new activities with their own activity type (and no longer log these to th... [23:09:38] eileen: i think this was the task with the previous queries: https://phabricator.wikimedia.org/T280587 [23:10:33] let me know if that doesn't look correct and i can dig further in phab. [23:12:26] looks like that is stored in crm/drupal/sites/default/civicrm/extensions/wmf-civicrm/Civi/Api4/Action/WMFDataManagement/ArchiveThankYou.php [23:14:26] dwisehaupt: yeah so we probably need to update that list as I assume our subject lines have changed over time [23:15:07] cool. yeah, and there is a p-c job that we can run or enable to fire it up. [23:15:29] then we can blast through it during the maint window. [23:15:52] i was a bit shocked at how quickly it has been approaching. [23:22:15] dwisehaupt: is there any way to log if truncate queries happen? wfan was looking at a phab to add a transaction to ensure clean rollback - but it looks like the rollback code is in place - so it would work unless there is a truncate happening [23:28:03] hmmmm... just truncates? [23:31:28] well truncates or create table both break transactions - but I don't think we are creating tables so I wondered if we truncate sometimes [23:31:55] i think the only way to get that is to turn on the query log which is all queries. [23:31:57] (although create table is used for temp tables & if without the temporary that breaks transactions too) [23:32:18] ah ok - I guess it *should* happen on staging [23:37:25] (03PS1) 10Eileen: Catch non-wmf-exceptions as errors [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/924606 (https://phabricator.wikimedia.org/T337550) [23:38:37] ejegg|food: when you are back see ^^ - I think the issue is bus errors not being recovered from - the bus error is a conflict over the file on disk - I have some thoughts about that but at the moment it is the failure to recover that has become an issue [23:39:49] note the specific contact did not have their thank you set - but now they do - https://civicrm.wikimedia.org/civicrm/contact/view?reset=1&cid=59635165&selectedChild=contribute - cos the next run sent it out [23:52:10] (03CR) 10Eileen: [C: 03+2] "Since my review was non-blocking I think this can go through now rather than let it hang around" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/921328 (https://phabricator.wikimedia.org/T334757) (owner: 10Damilare Adedoyin)