[00:03:37] <cstone>	 hey eileen im back from food and yeah I can since i merged that patch anyway
[00:04:12] <wikibugs>	 (03CR) 10Cstone: [C: 03+2] Update triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/853071 (https://phabricator.wikimedia.org/T322189) (owner: 10Eileen)
[00:05:21] <cstone>	 being so dark at 5pm is sad but I did acquire some lights for the balcony
[00:16:31] <wikibugs>	 (03Merged) 10jenkins-bot: Update triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/853071 (https://phabricator.wikimedia.org/T322189) (owner: 10Eileen)
[00:22:13] <wikibugs>	 (03PS3) 10Cstone: Change the Adyen audit file sort to look at modified date [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/852267 (https://phabricator.wikimedia.org/T321531)
[00:53:46] <eileen>	 ouch - today is like winter here  - I've had to put the heater on & my laptop has decided it is too cold to wake up
[00:58:30] <wikibugs>	 (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/854139
[00:58:43] <wikibugs>	 (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/854139 (owner: 10Eileen)
[01:05:20] <dwisehaupt>	 eileen: sounds like my dog. he has refused to do anything but sit in his bed or try to climb in my lap.
[01:35:31] <wikibugs>	 (03PS1) 10AndyRussG: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/854141
[01:36:48] <wikibugs>	 (03CR) 10AndyRussG: [C: 03+2] Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/854141 (owner: 10AndyRussG)
[01:39:23] <wikibugs>	 (03Merged) 10jenkins-bot: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/854141 (owner: 10AndyRussG)
[02:13:03] <wikibugs>	 (03Abandoned) 10Eileen: Damaged fixes - some may by php version specific but I should be on the same version as prod [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/852330 (https://phabricator.wikimedia.org/T295412) (owner: 10Eileen)
[02:14:11] <eileen>	 !log civicrm upgraded from 72fccce1 to b95f46bb
[02:14:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[02:14:30] <eileen>	 I put the heater on & my laptop woke up
[02:23:29] <cstone>	 I think I need to get a space heater
[02:27:46] <cstone>	 or my heat was randomly set to lower than I thought it was hha
[04:37:14] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:2) 2 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[04:42:14] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:3) 3 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[04:47:14] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:4) 4 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[04:52:14] <icinga-wm>	 RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[05:02:14] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:8) 8 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[05:07:14] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:8) 8 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[05:12:14] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[05:12:33] <dwisehaupt>	 looks like dlocal is having issues. checking their status page
[05:17:14] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:8) 8 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[05:17:44] <dwisehaupt>	 hmmm... looks like they may have popped in some new ips and this service is still hard coded vs using our tooling
[05:22:12] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[05:27:14] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:8) 8 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[05:32:14] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:8) 8 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[05:37:14] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:8) 8 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[05:42:14] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:8) 8 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[05:47:14] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[05:52:14] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[05:52:45] <wikibugs>	 (03PS2) 10AndyRussG: Move test ESI injection to BeforePageDisplay hook [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/854091 (https://phabricator.wikimedia.org/T320734)
[05:54:09] <wikibugs>	 10Fundraising-Backlog, 10fundraising-tech-ops: Add new dlocal API ip addresses - https://phabricator.wikimedia.org/T322611 (10Dwisehaupt)
[05:54:09] <eileen>	 dwisehaupt: FYI - I rolled out the new trigger sql onto live so they could be loaded at any suitable time
[05:54:09] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Move test ESI injection to BeforePageDisplay hook [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/854091 (https://phabricator.wikimedia.org/T320734) (owner: 10AndyRussG)
[05:55:00] <wikibugs>	 10Fundraising-Backlog, 10fundraising-tech-ops: Add new dlocal API ip addresses to firewall rules - https://phabricator.wikimedia.org/T322611 (10Dwisehaupt)
[05:55:11] <wikibugs>	 (03CR) 10AndyRussG: "recheck" [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/854091 (https://phabricator.wikimedia.org/T320734) (owner: 10AndyRussG)
[05:57:14] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[05:59:16] <dwisehaupt>	 eileen: cool thanks.
[06:00:32] <icinga-wm>	 ACKNOWLEDGEMENT - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] Dwisehaupt known. Bug: T322611 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[06:07:17] <wikibugs>	 10Fundraising Sprint Solid State Spaghetti, 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, and 3 others: CentralNotice: Add ESI test string to base HTML - https://phabricator.wikimedia.org/T320734...
[06:07:51] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_37) - 10https://gerrit.wikimedia.org/r/854205 (owner: 10L10n-bot)
[06:08:19] <wikibugs>	 10Fundraising-Backlog, 10fundraising-tech-ops: Add new dlocal API ip addresses to firewall rules - https://phabricator.wikimedia.org/T322611 (10Dwisehaupt) p:05Triage→03High a:03Dwisehaupt PFW deploy task: T322613
[06:09:36] <dwisehaupt>	 i'm not sure if i'm going to work tomorrow to avoid the election news or not. if i don't roll the triggers tomorrow, i'll try to do so on wednesday.
[07:01:48] <wikibugs>	 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising-Backlog: Adding and storing rml_source data for RML donations reporting - https://phabricator.wikimedia.org/T321163 (10MSuijkerbuijk_WMF) Hi @Eileenmcnaughton -- just wanted to check if there is an...
[07:32:14] <icinga-wm>	 RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages
[09:11:42] <wikibugs>	 (03CR) 10Nikerabbit: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_37) - 10https://gerrit.wikimedia.org/r/854205 (owner: 10L10n-bot)
[14:02:29] <wikibugs>	 (03PS4) 10Jgleeson: Change the Adyen audit file sort to look at modified date [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/852267 (https://phabricator.wikimedia.org/T321531) (owner: 10Cstone)
[14:04:45] <wikibugs>	 (03CR) 10Jgleeson: [C: 03+2] "This looks great. thanks for the extra comment too!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/852267 (https://phabricator.wikimedia.org/T321531) (owner: 10Cstone)
[14:18:00] <wikibugs>	 (03Merged) 10jenkins-bot: Change the Adyen audit file sort to look at modified date [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/852267 (https://phabricator.wikimedia.org/T321531) (owner: 10Cstone)
[14:20:00] <wikibugs>	 10Fundraising-Backlog, 10FR-Email: Click-tracking in Acoustic - https://phabricator.wikimedia.org/T316573 (10DBu-WMF) Hey all.  Any update on this click tracking.  I had hoped this would be fixed before en6C but now that we are in the throes of it it would be best to get it up as soon as possible.
[15:01:13] <wikibugs>	 10Fundraising Tech - Chaos Crew: Write unit tests for Damaged Extension - https://phabricator.wikimedia.org/T322640 (10Damilare)
[15:10:34] <wikibugs>	 10Fundraising Tech - Chaos Crew: Contact updated to have an invalid language - https://phabricator.wikimedia.org/T319142 (10Damilare) Hi @Eileenmcnaughton, we didn't really hit a new problem. The task was still on CC backlog and so I dug into it. Looks like Engage imports do not have a Language column set in the...
[15:17:05] <wikibugs>	 (03CR) 10Damilare Adedoyin: Return fallback language (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/854004 (https://phabricator.wikimedia.org/T319142) (owner: 10Damilare Adedoyin)
[17:46:44] <wikibugs>	 10Fundraising-Backlog, 10fundraising-tech-ops: Add new dlocal API ip addresses to firewall rules - https://phabricator.wikimedia.org/T322611 (10Dwisehaupt) 05Open→03Resolved Firewall rules deployed and connections working again.
[18:53:26] <wikibugs>	 (03PS2) 10Damilare Adedoyin: Return fallback language [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/854004 (https://phabricator.wikimedia.org/T319142)
[19:22:29] <wikibugs>	 (03PS1) 10Cstone: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/854577
[19:25:46] <wikibugs>	 (03CR) 10Cstone: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/854577 (owner: 10Cstone)
[19:27:14] <wikibugs>	 (03CR) 10Cstone: [C: 03+2] "recheck" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/854577 (owner: 10Cstone)
[19:38:49] <cstone>	 !log civicrm upgraded from b95f46bb to f2017495
[19:39:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:47:54] <wikibugs>	 (03PS1) 10Jgleeson: Update PayPal EC approvePayment/doExpressCheckoutPayment status mapping to handle all available status messages [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/854581 (https://phabricator.wikimedia.org/T318756)
[19:54:21] <wikibugs>	 (03PS2) 10Jgleeson: Update PayPal EC approvePayment/doExpressCheckoutPayment status mapping to handle all available status messages [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/854581 (https://phabricator.wikimedia.org/T318756)
[20:05:07] <wikibugs>	 (03PS3) 10Jgleeson: Update PayPal EC approvePayment/doExpressCheckoutPayment status mapping to handle all available status messages [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/854581 (https://phabricator.wikimedia.org/T318756)
[20:37:17] <jgleeson>	 gs
[20:49:38] <wikibugs>	 10Fundraising-Backlog: Add PaymentWarnings to SmashPig core (PayPal Warnings) - https://phabricator.wikimedia.org/T322692 (10jgleeson)
[20:53:40] <wikibugs>	 10Fundraising-Backlog: Add PaymentWarnings to SmashPig core (PayPal Warnings) - https://phabricator.wikimedia.org/T322692 (10jgleeson) Currently, we have three distinct types of errors in SmashPig, PaymentError, ValidationError and the newly proposed PaymentWarning. These types all share the property 'debugMessa...
[20:54:56] <wikibugs>	 (03PS1) 10Jgleeson: WIP: Add PaymentWarnings to SmashPig core. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/854612 (https://phabricator.wikimedia.org/T322692)
[20:55:30] <wikibugs>	 (03PS2) 10Jgleeson: WIP: Add PaymentWarnings to SmashPig core. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/854612 (https://phabricator.wikimedia.org/T322692)
[20:56:38] <wikibugs>	 (03PS4) 10Jgleeson: Update PayPal EC approvePayment/doExpressCheckoutPayment status mapping. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/854581 (https://phabricator.wikimedia.org/T318756)
[20:56:47] <wikibugs>	 (03PS3) 10Jgleeson: WIP: Add PaymentWarnings to SmashPig core. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/854612 (https://phabricator.wikimedia.org/T322692)
[21:28:33] <wikibugs>	 (03CR) 10Umherirrender: [C: 03+2] "Resubmit" [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/849705 (https://phabricator.wikimedia.org/T279431) (owner: 10Umherirrender)
[21:29:56] <wikibugs>	 10Fundraising Tech - Chaos Crew: Contact updated to have an invalid language - https://phabricator.wikimedia.org/T319142 (10Eileenmcnaughton) @Damilare ah that makes sense - I didn't look for NULL - I'll look again
[21:30:42] <wikibugs>	 (03Merged) 10jenkins-bot: Build: Use new unit-disallowed-list rule [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/849705 (https://phabricator.wikimedia.org/T279431) (owner: 10Umherirrender)
[22:17:13] <wikibugs>	 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising-Backlog, 10fundraising-tech-ops, 10SRE, and 3 others: SSL cert for links.email.wikimedia.org - https://phabricator.wikimedia.org/T188561 (10EWilfong_WMF) Thanks for the feedback and requirements...