[01:07:53] (03PS2) 10Ejegg: WIP update PendingDatabase for is_resolved [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1174076 (https://phabricator.wikimedia.org/T400304) [01:08:21] (03CR) 10CI reject: [V:04-1] WIP update PendingDatabase for is_resolved [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1174076 (https://phabricator.wikimedia.org/T400304) (owner: 10Ejegg) [01:09:09] (03PS3) 10Ejegg: WIP update PendingDatabase for is_resolved [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1174076 (https://phabricator.wikimedia.org/T400304) [01:48:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: ipset_error 14 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [01:53:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: ipset_error 19 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [01:58:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: ipset_error 19 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [02:03:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: ipset_error 19 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [02:08:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: ipset_error 20 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [02:13:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: ipset_error 19 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [02:18:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: ipset_error 20 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [02:23:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: ipset_error 19 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [02:28:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: ipset_error 19 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [02:33:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: ipset_error 17 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [02:38:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: ipset_error 18 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [02:43:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: ipset_error 18 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [02:48:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: ipset_error 8 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [02:53:18] RECOVERY - check_log_messages on frav1003 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [04:50:03] 06Fundraising-Backlog, 06FR-donorrelations, 07fr-current-sprint: PIX and Boleto after completing the donation do not show TY page - https://phabricator.wikimedia.org/T394979#11045715 (10RKumar_WMF) We would review this with DR team and get back. For now, deprioritising this and check in Q3. Thank you. [06:45:51] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_44) - 10https://gerrit.wikimedia.org/r/1174220 (owner: 10L10n-bot) [07:44:54] 06Fundraising-Backlog: RML Country information not making it into Civi from Acoustic - https://phabricator.wikimedia.org/T400786 (10krobinson) 03NEW [08:24:45] (03PS2) 10Hashar: tests: remove "mock" requirement [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1173945 (https://phabricator.wikimedia.org/T399304) [08:50:02] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_44) - 10https://gerrit.wikimedia.org/r/1174220 (owner: 10L10n-bot) [12:28:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:3) 3 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:33:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:4) 4 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:38:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:43:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:48:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:53:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:58:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:03:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:08:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:13:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:18:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:23:22] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:28:22] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:33:22] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:38:22] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:43:22] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:45:16] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 49.67, 55.43, 30.08 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [13:48:22] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:53:22] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:53:45] oy, lots of failmail overnight [13:54:01] looks like connection problems to dlocal? [13:58:21] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:01:40] oh i see, icinga is pinging this very channel about it [14:02:17] 06Fundraising-Backlog: RML Country information not making it into Civi from Acoustic - https://phabricator.wikimedia.org/T400786#11046934 (10SHust) One thing to note, I just came across this while investigating today: when I expand the address field in Civi, the country is there, but it doesn’t appear on the sum... [14:03:37] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:08:21] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:08:41] (03PS1) 10Wfan: add additional pending status for reject by fraud fail page [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1174470 (https://phabricator.wikimedia.org/T394098) [14:10:17] RECOVERY - check_load on frban2002 is OK: LOAD OK - total load average: 3.58, 3.68, 9.06 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:12:19] (03PS2) 10Wfan: add additional pending status for reject by fraud fail page [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1174470 (https://phabricator.wikimedia.org/T394098) [14:13:21] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:14:49] Jeff_Green: can you figure out what's going on with that dlocal endpoint check? [14:17:41] argh, somehow we're not requiring document for colombia now [14:17:47] and it's causing validation issues [14:17:51] lemme restore that [14:18:21] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:23:21] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:28:23] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:33:19] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:37:33] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-dlocal, 10MediaWiki-extensions-DonationInterface: DLocal CO donation form failing with no error message - https://phabricator.wikimedia.org/T400818 (10Ejegg) 03NEW [14:38:23] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:38:43] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-dlocal, 10MediaWiki-extensions-DonationInterface: DLocal CO donation form failing with no error message - https://phabricator.wikimedia.org/T400818#11047128 (10Ejegg) We should have tested after deployment, even if dlocal says they're allowing us... [14:43:21] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:48:21] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:53:05] (03PS1) 10Ejegg: Stop failing dlocal fiscal_number validation [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1174484 (https://phabricator.wikimedia.org/T395361) [14:53:21] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:53:36] (03PS2) 10Ejegg: Stop failing dlocal fiscal_number validation [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1174484 (https://phabricator.wikimedia.org/T400818) [14:58:08] ok, so are CO / CL / UY donations working via gravy? [14:58:23] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:01:13] well, CO just worked for me [15:01:39] so I guess the issue must just be from us to dlocal, not dlocal in general [15:03:01] seeing a few INR dlocal transactions in payments-initial but they're all failed. A lot more INR pageloads that don't seem to go anywhere [15:03:19] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:08:21] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:13:23] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:18:22] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:23:22] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:28:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:33:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:38:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:43:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:48:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:49:55] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Civi: multiple issues with recurring donations - https://phabricator.wikimedia.org/T399339#11047454 (10Ejegg) I'm trying to add all the affected users to this SearchKit: https://civicrm.wikimedia.org/civicrm/search#/display/Cancelle... [15:53:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:55:12] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add recipient to MGFE large donation bot emails - https://phabricator.wikimedia.org/T396791#11047487 (10AKanji-WMF) [15:58:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:03:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:03:38] (03CR) 10Wfan: [C:03+2] Stop failing dlocal fiscal_number validation [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1174484 (https://phabricator.wikimedia.org/T400818) (owner: 10Ejegg) [16:04:26] (03Merged) 10jenkins-bot: Stop failing dlocal fiscal_number validation [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1174484 (https://phabricator.wikimedia.org/T400818) (owner: 10Ejegg) [16:08:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:10:29] (03PS3) 10Ejegg: Log Gravy pending details so audit can read them [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1173980 (https://phabricator.wikimedia.org/T392156) [16:13:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:18:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:23:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:28:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:30:10] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Epc consumer with invalid data - https://phabricator.wikimedia.org/T398381#11047568 (10AnnWF) [16:33:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:38:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:43:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:48:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:53:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:54:49] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Settled Adyen Donation - Settled Gravy Donation - Not Appearing in Civi - https://phabricator.wikimedia.org/T395340#11047629 (10Ejegg) >>! In T395340#10926876, @EBrill-WMF wrote: > Hi Team, I have another example of this issue from... [16:58:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:59:51] (03PS1) 10Ejegg: Update SmashPig [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1174509 [16:59:52] (03CR) 10Ejegg: [C:03+2] Update SmashPig [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1174509 (owner: 10Ejegg) [17:03:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [17:08:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [17:13:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [17:18:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [17:20:52] (03Merged) 10jenkins-bot: Update SmashPig [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1174509 (owner: 10Ejegg) [17:23:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [17:26:10] (03CR) 10Ejegg: "resubmit" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1173980 (https://phabricator.wikimedia.org/T392156) (owner: 10Ejegg) [17:27:21] (03CR) 10Ejegg: "recheck" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1173980 (https://phabricator.wikimedia.org/T392156) (owner: 10Ejegg) [17:28:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [17:33:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [17:38:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [17:43:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [17:48:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [17:53:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [17:58:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [18:02:05] (03PS3) 10Wfan: Remove epc checksum timestamp valid check when consume [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1167965 (https://phabricator.wikimedia.org/T398381) [18:03:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [18:08:13] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [18:13:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [18:16:12] (03CR) 10Ejegg: [C:03+2] "resubmit" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1173980 (https://phabricator.wikimedia.org/T392156) (owner: 10Ejegg) [18:18:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [18:20:37] (03CR) 10CI reject: [V:04-1] Remove epc checksum timestamp valid check when consume [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1167965 (https://phabricator.wikimedia.org/T398381) (owner: 10Wfan) [18:20:48] fr-tech there's something odd happening with PayPal - looks like payments might be going through but are being failed on return [18:21:50] related to the payments-init failmails [18:23:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [18:28:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [18:33:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [18:38:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [18:39:25] (03Merged) 10jenkins-bot: Log Gravy pending details so audit can read them [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1173980 (https://phabricator.wikimedia.org/T392156) (owner: 10Ejegg) [18:43:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [18:43:31] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1174525 [18:43:33] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1174525 (owner: 10Ejegg) [18:44:50] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1174525 (owner: 10Ejegg) [18:48:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [18:53:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:6) 6 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [18:57:22] (03PS1) 10Ejegg: Update composer.lock for SmashPig upgrade [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1174529 [18:57:24] (03CR) 10Ejegg: [C:03+2] Update composer.lock for SmashPig upgrade [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1174529 (owner: 10Ejegg) [18:58:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [19:03:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [19:08:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [19:09:31] (03Merged) 10jenkins-bot: Update composer.lock for SmashPig upgrade [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1174529 (owner: 10Ejegg) [19:13:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [19:18:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [19:23:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [19:28:07] !log payments-wiki upgraded from 7de1798c to 0ab5bab9 [19:28:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:28:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [19:31:53] (03PS1) 10Eileen: Update moveContrib extension from 2.13 to 2.14 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174538 [19:32:11] (03CR) 10Ejegg: [C:03+2] Update moveContrib extension from 2.13 to 2.14 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174538 (owner: 10Eileen) [19:33:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [19:37:33] (03PS1) 10Eileen: Update nz.co.fuzion.extendedreport from 6.02 to 6.04 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174540 [19:38:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [19:43:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [19:48:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [19:51:57] (03CR) 10Eileen: [C:03+2] Point civi scripts at wmf not old wmff [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1172042 (owner: 10Ejegg) [19:52:25] (03CR) 10Eileen: [C:03+2] setup.sh --civicrm installs our standalone version [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1172041 (owner: 10Ejegg) [19:52:33] (03Merged) 10jenkins-bot: Update moveContrib extension from 2.13 to 2.14 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174538 (owner: 10Eileen) [19:52:37] (03CR) 10Wfan: [V:03+2] Point civi scripts at wmf not old wmff [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1172042 (owner: 10Ejegg) [19:53:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [19:58:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [19:59:30] (03PS2) 10Eileen: Update nz.co.fuzion.extendedreport from 6.02 to 6.04 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174540 [19:59:30] (03PS1) 10Eileen: Further update on movecontrib [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174542 [19:59:30] (03PS1) 10Eileen: Update Contact editor from 2.6 to 2.7 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174543 [19:59:30] (03PS1) 10Eileen: Update geocoder [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174544 [19:59:31] (03PS1) 10Eileen: Update Contact Layout editor to 3.0 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174545 [20:03:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [20:08:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [20:13:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [20:18:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [20:23:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [20:28:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [20:33:14] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [20:35:27] ACKNOWLEDGEMENT - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Astropay-DLocal:7) 7 [=1] Jeff_Green known astropay issue https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [20:39:10] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-PayPal-ExpressCheckout: Intermittent PayPal auth failures lead to bad payments-initial messages - https://phabricator.wikimedia.org/T400846 (10Ejegg) 03NEW [21:01:13] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Civi: multiple issues with recurring donations - https://phabricator.wikimedia.org/T399339#11048370 (10Ejegg) OK, I've been able to restrict it to just the google / apple pay ones and have added the restriction to just include peopl... [21:03:33] (03PS1) 10Eileen: Remove some fields from messages tests that are not actually in use [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174554 [21:05:56] (03CR) 10Ejegg: [V:03+2] setup.sh --civicrm installs our standalone version [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1172041 (owner: 10Ejegg) [21:18:40] (03CR) 10CI reject: [V:04-1] Remove some fields from messages tests that are not actually in use [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174554 (owner: 10Eileen) [21:29:49] (03PS2) 10Eileen: Remove some fields from messages tests that are not actually in use [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174554 [21:37:02] (03PS5) 10Eileen: First cut at WMFAudit::settle api [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170226 (https://phabricator.wikimedia.org/T396071) [21:37:02] (03PS1) 10Eileen: WIP Add audit action [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174561 [21:57:21] (03CR) 10CI reject: [V:04-1] WIP Add audit action [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1174561 (owner: 10Eileen) [22:02:15] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Fix redis related bug in local dev environment - https://phabricator.wikimedia.org/T334036#11048511 (10Eileenmcnaughton) 05Open→03Resolved a:03Eileenmcnaughton [22:02:55] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Civi: multiple issues with recurring donations - https://phabricator.wikimedia.org/T399339#11048517 (10SHust) A huge thanks, Elliott! Separating those who received the recurring failure email from those who didn’t was the blocker we... [22:08:07] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Review Redis session handler locking - https://phabricator.wikimedia.org/T400848 (10Eileenmcnaughton) 03NEW [22:17:26] PROBLEM - check_trino on fransw2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_trino [22:22:26] PROBLEM - check_trino on fransw2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_trino [22:27:26] PROBLEM - check_trino on fransw2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_trino [22:32:27] PROBLEM - check_trino on fransw2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_trino [22:37:19] RECOVERY - check_trino on fransw2001 is OK: HTTP OK: HTTP/1.1 200 OK - 2450 bytes in 3.262 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_trino [22:38:53] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Wikimedia-Fundraising-CiviCRM: Review Redis session handler locking - https://phabricator.wikimedia.org/T400848#11048591 (10Eileenmcnaughton) [23:21:35] 03Fundraising Sprint: Omakase, 06Fundraising-Backlog, 07fr-current-sprint, 07payments-orchestration: Discuss how we will handle the banking descriptor limit - https://phabricator.wikimedia.org/T398652#11048622 (10EMartin) [23:27:25] PROBLEM - check_trino on fransw2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_trino [23:32:15] PROBLEM - check_trino on fransw2001 is CRITICAL: connect to address 127.0.0.1 and port 8443: Connection refused https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_trino [23:37:07] PROBLEM - check_load on fransw2001 is CRITICAL: LOAD CRITICAL - total load average: 5.01, 84.04, 58.91 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_load [23:37:15] PROBLEM - check_trino on fransw2001 is CRITICAL: connect to address 127.0.0.1 and port 8443: Connection refused https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_trino [23:42:07] RECOVERY - check_load on fransw2001 is OK: LOAD OK - total load average: 0.73, 31.18, 42.79 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_load [23:42:15] RECOVERY - check_trino on fransw2001 is OK: HTTP OK: HTTP/1.1 200 OK - 836 bytes in 0.107 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_trino