[00:17:52] (03PS1) 10Eileen: Stock 5.45rc CiviCRM [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/751847 (https://phabricator.wikimedia.org/T298664) [00:18:24] (03PS1) 10Eileen: Add end of year thank you template and example code [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/751848 (https://phabricator.wikimedia.org/T290253) [00:18:55] (03PS1) 10Eileen: Fixes that still need upstreaming [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/751849 [00:30:38] (03CR) 10jerkins-bot: [V: 04-1] Add end of year thank you template and example code [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/751848 (https://phabricator.wikimedia.org/T290253) (owner: 10Eileen) [00:30:40] (03CR) 10jerkins-bot: [V: 04-1] Fixes that still need upstreaming [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/751849 (owner: 10Eileen) [00:31:11] (03CR) 10jerkins-bot: [V: 04-1] Stock 5.45rc CiviCRM [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/751847 (https://phabricator.wikimedia.org/T298664) (owner: 10Eileen) [01:49:17] hmm, more EOY failmail? [01:51:25] ah, the new fix just changes the error message [01:52:06] manually fixed [02:07:48] ejegg: arg - it didn;t get caught :-( [02:09:11] eileen: heh, at least it was quicker to do the manual update with that error message [02:09:20] :-) [02:15:57] (03PS1) 10Eileen: Catch render exception [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751853 [02:15:58] ejegg: ^^ [02:16:25] (03PS2) 10Eileen: Catch render exception [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751853 [02:16:35] looking [02:17:50] (03CR) 10Ejegg: [C: 03+2] Catch render exception [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751853 (owner: 10Eileen) [02:29:44] (03Merged) 10jenkins-bot: Catch render exception [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751853 (owner: 10Eileen) [02:30:50] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/751854 [02:31:03] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/751854 (owner: 10Eileen) [02:34:42] !log civicrm revision changed from 67264062 to 3d334f30 [02:34:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:37:23] (03PS1) 10Eileen: Very temporary addition of useTrash [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/751856 [03:47:09] is anyone is about & able to merge ^^ then it would help me to see if I can make unit test changes for upcoming civi changes [04:03:02] (03PS2) 10Eileen: Fixes that still need upstreaming [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/751849 [04:14:21] (03CR) 10jerkins-bot: [V: 04-1] Fixes that still need upstreaming [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/751849 (owner: 10Eileen) [04:51:48] (03PS1) 10Eileen: Fix text only version of eoy_template to not break some tokens [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751857 (https://phabricator.wikimedia.org/T297159) [04:52:05] is anyone about [04:59:46] 10Fundraising Sprint Visual C Saw, 10Fundraising Sprint Xenomorph Petting Zoo, 10Fundraising-Backlog, 10fundraising sprint Universal Cereal Bus, and 5 others: Switch eoy email over fully to the new method - https://phabricator.wikimedia.org/T297159 (10Eileenmcnaughton) I started looking at T298664 and appl... [05:01:52] if anyone is about I sent an email & updated ^^ [05:15:39] I also notified on slack [12:07:11] g'day fr-tech. 358685 sent, 300957 queued and 27 failed. going good! [12:08:04] past the half way point! [12:10:53] hmm let's check that EOY failmail [12:53:18] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 7616 7500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 7 keys, up 60 days 11 hours - memory use is 6.35M (peak 9.13M, 0.13% of max, fragmentation 1.73%), connected_slaves is 3, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 74, payments-antifraud is 14, payments-init is 9, pending is 2, refund is 0, unsubscribe is 0 https://icinga.wikimedi [12:53:18] -bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [12:55:26] shoot looks like logging in with lowercase first initial also doesn't fix the icinga issue [12:55:37] I just tried to ACK that alert fr-tech but it still fails [12:55:39] :( [12:58:18] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 7813 7500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 60 days 11 hours - memory use is 6.39M (peak 9.13M, 0.13% of max, fragmentation 1.70%), connected_slaves is 3, donations is 2, jobs is 0, jobs-adyen is 1, jobs-paypal is 0, payments-antifraud is 15, payments-init is 107, pending is 1, refund is 0, unsubscribe is 0 https://icinga.wikimed [12:58:18] i-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [13:05:56] (03CR) 10Jgleeson: "This looks good to me and kinda makes sense :) apart from the bold tags transforming to uppercase, could you elaborate a bit more on that?" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751857 (https://phabricator.wikimedia.org/T297159) (owner: 10Eileen) [13:09:03] 10Fundraising-Backlog, 10SRE: Fundraising-Tech engineers unable to ACK icinga alerts on fr-tech host groups - https://phabricator.wikimedia.org/T298649 (10jgleeson) Thanks @Dzahn I've just tried ACKing another alert after logging in with all lower case chars but the outcome is still the same sadly. {F34909313} [13:23:18] RECOVERY - check_redis on frqueue1003 is OK: OK: REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 6 keys, up 60 days 12 hours - memory use is 5.53M (peak 9.13M, 0.12% of max, fragmentation 1.84%), connected_slaves is 3, donations is 1, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 5, payments-init is 0, pending is 0, recurring is 6263, refund is 0, unsubscribe is 2 https://icinga.wikimedia.org/cgi-bin/icing [13:23:18] .cgi?type=2&host=frqueue1003&service=check_redis [15:17:11] PROBLEM - check_load on frdb1003 is CRITICAL: CRITICAL - load average: 28.32, 25.90, 15.72 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_load [15:22:11] PROBLEM - check_load on frdb1003 is CRITICAL: CRITICAL - load average: 40.26, 33.23, 21.51 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_load [15:27:11] PROBLEM - check_load on frdb1003 is CRITICAL: CRITICAL - load average: 8.04, 22.84, 20.97 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_load [15:42:11] RECOVERY - check_load on frdb1003 is OK: OK - load average: 6.27, 3.27, 8.94 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_load [16:25:57] (03CR) 10Jgleeson: "Just for reference this is the raw EOY email I received earlier https://phab.wmfusercontent.org/file/data/hkurfol3ywz67zu7gw37/PHID-FILE-3" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751857 (https://phabricator.wikimedia.org/T297159) (owner: 10Eileen) [16:59:09] morning folks. [16:59:19] or afternoon, or evening. [17:00:23] morning [17:00:37] i was wondering if there is a way we could get a rough idea of the number of recurrings that are expected each day (including collapsed days like Feb 28). the idea being we could adjust our alerting thresholds so we don't page for known large recurring runs. [17:03:48] sounds like a good idea dwisehaupt [17:06:06] morning also! [18:55:36] 10Fundraising-Backlog, 10SRE: Fundraising-Tech engineers unable to ACK icinga alerts on fr-tech host groups - https://phabricator.wikimedia.org/T298649 (10Dzahn) a:03Dzahn [18:55:45] 10Fundraising-Backlog, 10SRE: Fundraising-Tech engineers unable to ACK icinga alerts on fr-tech host groups - https://phabricator.wikimedia.org/T298649 (10Dzahn) 05Open→03In progress [19:18:49] 10Fundraising-Backlog, 10SRE, 10Patch-For-Review: Fundraising-Tech engineers unable to ACK icinga alerts on fr-tech host groups - https://phabricator.wikimedia.org/T298649 (10Dzahn) Hey @jgleeson soo.. you did not have an Icinga contact and that had to be created in the private puppet repository. I just did... [19:19:25] 10Fundraising-Backlog, 10SRE, 10observability, 10serviceops-radar, 10Patch-For-Review: Fundraising-Tech engineers unable to ACK icinga alerts on fr-tech host groups - https://phabricator.wikimedia.org/T298649 (10Dzahn) [19:22:14] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [19:27:14] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [19:40:38] 10Fundraising-Backlog, 10SRE, 10observability, 10serviceops-radar, 10Patch-For-Review: Fundraising-Tech engineers unable to ACK icinga alerts on fr-tech host groups - https://phabricator.wikimedia.org/T298649 (10Jgreen) >>! In T298649#7602585, @Dzahn wrote: > > When I did a " grep contact_groups puppet_h... [19:42:14] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [19:47:14] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [20:13:38] (03CR) 10Eileen: "jgleeson - it's the line below the comment block that does the transform to html - & that process does the upper casing." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751857 (https://phabricator.wikimedia.org/T297159) (owner: 10Eileen) [20:55:39] thanks eileen. any idea why we're transforming text in bold html tags to uppercase? [20:55:58] or is that just something that's happening in the email realms for a while? [20:56:17] jgleeson: so the function that strips out the html (from roundcube) replaces them with something if it sees fit [20:56:25] ie

goes to line breaks [20:56:45] for bold it switches to upper case because that is a text only way to make it stand out [20:57:07] ahhh ok that makes sense [20:57:38] yeah - when you think about if from the right end of the process it does but it's a bit whoa when you first hit it [20:57:41] so it tries to do a non-html equivalent modification [20:58:02] yeah [20:58:05] yep [20:58:22] ok cool now I understand. thanks for the explanation! [20:58:34] I hit that bug because I tried to throw the new civi version at jenkins [20:58:58] & it turned notice grumpiness back on in our dev & staging [20:59:04] dev & CI I mean [20:59:15] & started grumping about a notice on YEAR [20:59:23] but - your email seems to have rendered it? [21:00:07] if you look at the text version here https://civicrm.wikimedia.org/civicrm/admin/messageTemplates?reset=1#/edit?id=310 [21:00:21] you will see year in caps - if just needs to be edited to lcase in the UI [21:00:42] but I felt queasy about doing that while emails are going out [21:01:55] the text around year in the text content [21:02:04] yeah I see all that in uppercase [21:02:24] yep - I think it wasn't in your email because your text version might have been from your mail client? [21:02:55] that was the o"riginal message" in gmail lingo [21:03:07] or perhaps some other process generated it later on ... I wasn't expecting that since if you preview at the link ^^ it shows the odd thing [21:03:51] how do I preview? [21:04:23] expand the text & then look on the right for the little eye [21:04:45] oh yeah I see it [21:05:03] hmm yeah so looks like the preview isn't rendering the $vars [21:05:13] or those ones at least [21:05:20] yeah - specifically those [21:05:44] but based on what you received it must be thrown away & some other process is converting the html email later on [21:06:05] oh interesting [21:07:07] jgleeson: also - can you +2 this https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/civicrm/+/751856 - [21:07:16] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [21:07:42] looking [21:07:48] I can't update the tests for 5.45 without a dummy property so the set methods will work (well I could add painful checks but this seems self obsoleting) [21:08:02] one trick wonder property lol [21:08:52] (03CR) 10Jgleeson: [C: 03+2] "LGTM :)" [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/751856 (owner: 10Eileen) [21:09:08] (03PS1) 10Eileen: Ensure full deletion after upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751996 [21:09:15] jgleeson: yep & then ^^ will be able to pass [21:09:54] eileen: I'm curious about the other patch now. Is there actually a problem outside of dev & CI if I'm getting the right text version rendered [21:10:27] or is this possibly some kind of env discrepancy [21:11:19] jgleeson: I think not now - check EOYEmail/Render [21:11:40] it actually ONLY returns the html version 'html' => str_replace('

', '', $rendered['html']), [21:12:16] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [21:17:17] it's at this point I'm wishing I'd done some review of this code so I'd have xdebugged through what happens in Civi\Api4\WorkflowMessage::render [21:17:37] and have an idea how the html key gets set [21:18:38] I guess there's too many gaps in my head about how all this works to comment eileen :) [21:19:28] jgleeson: well when I thought there was a bug I was assuming that BOTH the html version AND the text version would be returned at that point. Then both would be sent out [21:19:59] since only the html version is returned it must be that the text version is generated elsewhere - likely by phpmailer - and that it is based off the rendered html version [21:20:32] ah i see [21:20:58] (03Merged) 10jenkins-bot: Very temporary addition of useTrash [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/751856 (owner: 10Eileen) [21:21:01] presumably just using strip tags [21:21:28] (03CR) 10jerkins-bot: [V: 04-1] Ensure full deletion after upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751996 (owner: 10Eileen) [21:22:09] jgleeson: yep likely [21:22:36] Also fyi - you can see https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/751996 failed because it ran before that other patch was merged so you can see the fail / issue it was addressing [21:25:16] I sees I sees [21:26:32] i can probably +2 that then before jenkins has another go [21:27:38] thanks jgleeson [21:30:21] eileen: does that patch point to the latest merged civicrm symlink? [21:31:01] ah looks like it does [21:31:03] https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/fundraising/crm/+/be4dcd020e4dedb088d4f081db4c370e5f67250f/civicrm [21:31:54] jgleeson: hmm - I do need to update the submodule I guess [21:32:16] (03CR) 10Jgleeson: [C: 03+2] "I've looked into the future and I'm fairly certain jenkins will +2 this patch shortly after I do. An oracle can never always know, but the" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751996 (owner: 10Eileen) [21:32:27] lol [21:33:13] (03PS1) 10Eileen: CiviCRM submodule commit [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751998 [21:33:16] I was watching matrix clips on youtube over christmas. couldn't help myself :) [21:33:25] (03CR) 10Eileen: [C: 03+2] CiviCRM submodule commit [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751998 (owner: 10Eileen) [21:33:33] hah [21:35:01] https://youtu.be/qTnBVDKuNdI?t=227 [21:36:32] lol [21:37:13] jenkins will -1 the patch now to spite me [21:37:20] :) [21:39:11] I'm heading off shortly eileen but not sure if we need to +2 the capsy bold patch anymore? [21:39:28] jgleeson: yeah - for tests I still need it merged [21:39:36] that patch only ever affected dev [21:40:07] gotcha [21:42:04] 10Fundraising-Backlog, 10SRE, 10observability, 10serviceops-radar: Fundraising-Tech engineers unable to ACK icinga alerts on fr-tech host groups - https://phabricator.wikimedia.org/T298649 (10Dzahn) >>! In T298649#7602627, @Jgreen wrote: > Fundraising-related services are done with passive checks, so the r... [21:43:41] (03CR) 10Jgleeson: [C: 03+2] "Thanks! we talked this through on IRC and it fixes the issue." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751857 (https://phabricator.wikimedia.org/T297159) (owner: 10Eileen) [21:44:10] thanks jgleeson [21:44:40] (03Merged) 10jenkins-bot: Ensure full deletion after upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751996 (owner: 10Eileen) [21:45:07] (03Merged) 10jenkins-bot: CiviCRM submodule commit [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751998 (owner: 10Eileen) [21:47:00] 10Fundraising-Backlog, 10SRE, 10observability, 10serviceops-radar: Fundraising-Tech engineers unable to ACK icinga alerts on fr-tech host groups - https://phabricator.wikimedia.org/T298649 (10jgleeson) Thanks for all the digging on this @Dzahn, hugely appreciated! Sorry to be a pain but a few others on fr... [21:49:32] eileen: anything else before I shoot? [21:49:46] low hanging fruit ish [21:50:21] jgleeson: no - there is a UI fix for deduper but that is not blocking me on the tests so can wait another day [21:50:32] sounds good [21:50:42] have a good rest of your day fr-tech. bye for now [21:50:45] have a good evening (well night now!) [21:50:52] you too! [21:55:47] (03Merged) 10jenkins-bot: Fix text only version of eoy_template to not break some tokens [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/751857 (https://phabricator.wikimedia.org/T297159) (owner: 10Eileen) [22:00:22] 10Fundraising-Backlog, 10SRE, 10observability, 10serviceops-radar, 10Patch-For-Review: Fundraising-Tech engineers unable to ACK icinga alerts on fr-tech host groups - https://phabricator.wikimedia.org/T298649 (10Dzahn) @jgleeson Let's see if it works now, as "jgleeson". it should work both via global rig... [22:02:45] 10Fundraising-Backlog, 10SRE, 10observability, 10serviceops-radar, 10Patch-For-Review: Fundraising-Tech engineers unable to ACK icinga alerts on fr-tech host groups - https://phabricator.wikimedia.org/T298649 (10Dzahn) Would still have to confirm whether it's actually enough to use only the fr-tech-ops g...