[15:07:16] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [15:12:16] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [15:43:24] looks like the lat two EOY jobs have failed let's see what's up [15:47:26] ok looks like an exception was thrown for a contact having no donations in the year we're sending for [15:47:56] which is odd because I thought we we're sending to folks we've already checked had donated [15:48:20] I'm sure there's an explanation but let's mark this contact as failed for now to get things moving [15:48:46] fr-tech I'm reading the logs files included in the latest failmails [15:49:03] luckily the contact ID is output in the exception so it should be easy enough to fix [15:49:18] just need to find the email for for that donor [15:50:19] I think we have a linked table in civi called something like civicrm_email which has a foreign key for contact [15:52:06] ok got the email [15:53:15] so now I need to run a query on the live db using the `drush civicrm-sql-cli` terminal [16:01:37] ok the affected donor record is now set to 'failed' hopefully the next job which I think runs at 4m on the hour will kick in [16:06:27] ok it looks like the job has started and is running [16:08:08] fr-tech as this exception effectively stops the EOY sends I'm gonna add some notes to the EOY notes doc so that others can fix it if it comes up again [16:08:24] Sounds good jgleeson [16:14:02] oh hey jgleeson just getting online here [16:14:25] jgleeson: how many are remaining? [16:16:21] Hey dstrine, damilare: wanna get a count for dstrine whilst I finish up these notes? [16:16:47] ok one min [16:16:59] thanks much! [16:38:08] 10Fundraising Sprint Visual C Saw, 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 3 others: Question about blocked contacts in Civi and how they sync to Acoustic - https://phabricator.wikimedia.org/T293587 (10nisrael) Hi Eileen, thanks for sharing th... [16:39:03] ok fr-tech I've added some notes to the EOY notes doc on how to fix the ‘No contributions in the given year’ exceptions. It has a nice big red heading https://docs.google.com/document/d/1vU8ZSrZHim7qVTi9aEkakptUHs9283DTakdx0abOXmQ/edit [16:59:03] Thanks, very helpful, and then we should find out why our filter missed those two donors with no donation in year 2021 right as a task right? [17:00:41] definitely WFan. that needs following up [17:00:48] wanna create that task? [17:10:01] jgleeson: Haha enjoy your dinner. I will watch the Phabricator to see if this task been created :) [18:12:59] 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Fix banner bump with server-side, cache-layer banner selection - https://phabricator.wikimedia.org/T283521 (10AndyRussG) [18:29:52] fr-tech looks like more failmail [18:30:02] let's see what it is [18:32:15] ahhh it's the same error which is blocking the jobs [18:32:22] they really don't wanna finish today do they [18:33:25] ahhh nostalgic EoY e-mails [18:33:48] ok let's find the affected record and fix it so we can close the week out [18:33:58] indeed AndyRussG! [18:33:59] nostalgic for their childhood in the calculation phase on the server, hesitant about leaving home [18:34:06] ha [18:34:11] I'm about if I can help btw [18:34:42] AndyRussG: I was gonna suggest maybe someone else fix it to spread the knowledge on this but maybe in this instance we wanna it switched bac on ASAP [18:34:46] +k [18:35:13] if you have a quick fix handy and would like to go ahead, pls don't hesistate to do so :) [18:35:38] AndyRussG: I've added notes on the fix I used today here https://docs.google.com/document/d/1vU8ZSrZHim7qVTi9aEkakptUHs9283DTakdx0abOXmQ/edit [18:35:55] ah fantastic thx! [18:36:04] figured it would be handy to have documented as we've got no eileen or ejegg today i think [18:36:40] ahh right [18:37:52] we've got about 12 minutes until the next job runs. wanna jump on a call and talk through fixing it to switch things back on? [18:37:55] AndyRussG: ^ [18:37:59] fr-tech^ [18:38:11] sure one sec! [18:40:15] ok folks I'll jump on today's standup meet link and get prepped [18:40:24] https://meet.google.com/nsc-symi-ici?authuser=0&hs=122 [19:25:35] I'm calling it a day fr-tech. The emails are in a good place and AndyRussG is on top of things in the case of any more shenanigans! Have a good weekend all and bye for now :) [19:46:44] 10Fundraising-Backlog: EOY failmail - ‘No contributions in the given year’ exceptions blocking the mail sends job - https://phabricator.wikimedia.org/T298803 (10AnnWF) [20:18:54] 10Fundraising-Backlog, 10SRE, 10observability, 10serviceops-radar: Fundraising-Tech engineers unable to ACK icinga alerts on fr-tech host groups - https://phabricator.wikimedia.org/T298649 (10Dzahn) a:05Dzahn→03jgleeson [20:19:43] 10Fundraising-Backlog, 10SRE, 10observability, 10serviceops-radar: Fundraising-Tech engineers unable to ACK icinga alerts on fr-tech host groups - https://phabricator.wikimedia.org/T298649 (10Dzahn) @jgleeson We can either resolve this if it works for you or keep using it for the other people that need to... [20:44:16] 10Fundraising-Backlog, 10SRE, 10observability, 10serviceops-radar: Fundraising-Tech engineers unable to ACK icinga alerts on fr-tech host groups - https://phabricator.wikimedia.org/T298649 (10jgleeson) Sorry Dan I forgot to check in on this today and have finished work as I'm working from the UK. I'll tes... [20:46:00] 10Fundraising-Backlog, 10SRE, 10observability, 10serviceops-radar: Fundraising-Tech engineers unable to ACK icinga alerts on fr-tech host groups - https://phabricator.wikimedia.org/T298649 (10Dzahn) Yes yes, there was no expectation that this happens right now or you work on the weekend. This was for Monda... [20:46:31] 10Fundraising-Backlog, 10SRE, 10observability, 10serviceops-radar: Fundraising-Tech engineers unable to ACK icinga alerts on fr-tech host groups - https://phabricator.wikimedia.org/T298649 (10Dzahn) p:05Triage→03Medium [21:28:06] seems contact ID 54569961 FAILED the same prob [21:29:47] AndyRussG: do you want me to do the job jack did this morning or you are working on it? [21:31:43] WFan: oh oops I hadn't seen it yet :) would you like to go ahead and fix that one? [21:31:59] Ok [21:32:11] thanks so much WFan! [21:33:39] WFan: If you'd like me to follow along on a screenshare, pls don't hesitate to let me know... same also if you have questions... also I'm sure you can do it of course!!! :) [21:38:33] the email address belongs to contact id 54569961 is marked as failed, I am checking if the job is running again now [21:41:41] Thanks WFan [21:42:36] woohoo thanks so much WFan! :) [21:49:19] Np AndyRussG hope this will be the last one with this error~ [21:51:09] :) [21:51:51] 10Fundraising-Backlog: EOY failmail - ‘No contributions in the given year’ exceptions blocking the mail sends job - https://phabricator.wikimedia.org/T298803 (10AnnWF) [21:53:24] The issue with the no contributions bug is that it is a different exception than we are catching - I"m just trying to decide whether to catch the generic thrown exception (no contribution) or create a new one - hmm might have just decided on the latter but not wavering again [21:54:45] I guess for EOY mails we now have no queue [21:55:17] WFan: aha - but I can change the status of the failed ones back to queued so it's a great time to test a fix :-) [21:55:28] The idea is that it should re-fail them gracefully [21:56:58] eileen: haha right, and the same issue for the no contribution will pop up again :) [21:59:10] (03PS1) 10Eileen: Add exception for no valid contribution scenario [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/752221 (https://phabricator.wikimedia.org/T298803) [21:59:45] yep - WFan unfortunately because CiviCRM is in a submodule we have to some commit juggling so ^^ will create the exception & add catching for it [22:00:04] but we can only update the civicrm repo to throw that exception type once ^^ is merged [22:01:33] 👍 [22:02:18] - an added piece of the puzzle is that the code that throws it 'shouldn't really' be in the civicrm repo - but it is there for now because of a temporarily missing piece of functionality (the ability to load workflow templates from extensions - it's actually by-design because the workflow template code is new & from a core pov want to do some more bedding down before supporting it to work outside core - it was [22:02:19] developed for us so we are the only users of the custom workflow classes so far) [22:03:30] (03PS1) 10Eileen: Use NoContributionException so it is caught [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/752222 (https://phabricator.wikimedia.org/T298803) [22:03:46] ^^ is the fix to use the exception (but it will fail until the first is merged) [22:07:43] !log config revision changed from 3df415c1 to ecf09aa0 - disable eoy email jobs [22:07:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:16:45] eileen: https://phabricator.wikimedia.org/T298803 this is the task I created for this problem you just commit :) [22:18:09] WFan: oddly I referenced that in the commit message but the patch hasn't been linked from the phab [22:24:13] hmm wired, haha I did see your commit message has this bug id [22:26:45] 10Fundraising-Backlog: EOY failmail - ‘No contributions in the given year’ exceptions blocking the mail sends job - https://phabricator.wikimedia.org/T298803 (10AnnWF) Eileen updated the error exception to NoContributionException instead of CRM_Core_Exception https://gerrit.wikimedia.org/r/c/wikimedia/fundraisin... [22:30:07] 10Fundraising-Backlog: EOY failmail - ‘No contributions in the given year’ exceptions blocking the mail sends job - https://phabricator.wikimedia.org/T298803 (10Eileenmcnaughton) Note we ALSO need this patch https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/752221 [22:34:52] 10Fundraising-Backlog, 10fr-donorservices: Featured Request: Have Civi reflect when an email has been unsubscribed 90 days - https://phabricator.wikimedia.org/T296054 (10SHust) Hi team, please let me know if there's anything I can do to get this rolling in Q3. Thank you! [23:16:06] 10Fundraising-Backlog, 10fr-donorservices: Featured Request: Have Civi reflect when an email has been unsubscribed 90 days - https://phabricator.wikimedia.org/T296054 (10bsisolak) Something struck me on how you can get this data. If you run this API call each day: