[02:54:14] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Already cancelled paypals are getting cancelled again - https://phabricator.wikimedia.org/T367623#9905806 (10Cstone) This is fixed, the stats were delayed a day as the fix didn't touch the ones that already got into the failing state and then were cancell... [03:01:03] 03Fundraising Sprint: LeeroyJenkinsssss(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2021, 13Patch-For-Review: Move wmf_audit logic into wmf-civicrm Civi extension - https://phabricator.wikimedia.org/T270675#9905811 (10Cstone) [03:02:54] 03Fundraising Sprint: LeeroyJenkinsssss(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2021, 13Patch-For-Review: Move wmf_audit logic into wmf-civicrm Civi extension - https://phabricator.wikimedia.org/T270675#9905812 (10Cstone) [03:59:55] (03PS1) 10Cstone: WIP: Fix paypal cancellation dates [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1047215 (https://phabricator.wikimedia.org/T367623) [05:26:19] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_40) - 10https://gerrit.wikimedia.org/r/1046847 (owner: 10L10n-bot) [08:52:12] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] Use namespaced CLDR classes [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1040758 (owner: 10Umherirrender) [09:48:19] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [09:53:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 36 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [09:58:19] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 36 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [10:03:25] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 32 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [10:08:21] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 32 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [10:13:19] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 33 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [10:18:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [10:23:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [10:28:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [10:33:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [10:38:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [10:43:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [10:48:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [10:53:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [10:58:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 33 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:03:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:08:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:13:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:18:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:23:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 36 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:28:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:33:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 33 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:38:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:43:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 36 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:48:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:53:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 33 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [11:58:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 37 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:03:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 33 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:08:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 37 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:13:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:18:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 37 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:20:34] (03PS18) 10Damilare Adedoyin: Create gravy special page and load components [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1040198 (https://phabricator.wikimedia.org/T364508) [12:22:37] (03CR) 10CI reject: [V:04-1] Create gravy special page and load components [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1040198 (https://phabricator.wikimedia.org/T364508) (owner: 10Damilare Adedoyin) [12:23:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:28:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:33:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:33:28] (03PS1) 10Damilare Adedoyin: Add recurring payment token to data sent to queue [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1047493 (https://phabricator.wikimedia.org/T367578) [12:35:29] (03CR) 10CI reject: [V:04-1] Add recurring payment token to data sent to queue [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1047493 (https://phabricator.wikimedia.org/T367578) (owner: 10Damilare Adedoyin) [12:38:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 33 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:43:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:45:08] fr-tech. we don't use ingenico anymore so I think we can ignore/suppress the spam [12:48:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:53:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 36 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [12:58:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:03:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:23:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:28:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 33 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:33:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:38:12] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 36 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:43:16] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 31 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:48:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 36 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:53:18] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6303 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [13:53:18] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 36 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [13:58:15] RECOVERY - check_mysql on frdb1004 is OK: Uptime: 410449 Threads: 4 Questions: 20439832 Slow queries: 193 Opens: 1189 Open tables: 1123 Queries per second avg: 49.798 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [13:58:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:03:27] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:08:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:13:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:18:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:19:01] I just tried Acking the alerts again but I'm not authorized [14:23:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:28:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:29:59] 06Fundraising-Backlog, 10FR-Email: New columns to add to Civi DatabaseUpdate export in July - https://phabricator.wikimedia.org/T366529#9907200 (10Eileenmcnaughton) I'm not familiar with the last 2 - are they recently added fields? [14:31:50] 06Fundraising-Backlog, 10FR-Email: New columns to add to Civi DatabaseUpdate export in July - https://phabricator.wikimedia.org/T366529#9907208 (10Eileenmcnaughton) actually now I look at it - several of these are not existing CiviCRM fields so it might take some work to figure out how to calculate them - howe... [14:33:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:38:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 36 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:43:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:48:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:53:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 37 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [14:58:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:7) 7 [=1], ipset_error 32 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:03:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:05:01] 06Fundraising-Backlog, 10FR-donorservices: Endowment 'Thank You' email/receipt sent in English to non-English donors - https://phabricator.wikimedia.org/T367594#9907297 (10CDenes_WMF) Hello. Until now we have always been told to keep everything Endowment related to English-only. As far as I know, we don't do... [15:08:13] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:13:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:18:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:23:13] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 32 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:24:33] (03CR) 10Damilare Adedoyin: [C:03+2] Auto-cancel PayPal subscriptions on chargeback [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1038400 (https://phabricator.wikimedia.org/T361243) (owner: 10Ejegg) [15:27:04] (03CR) 10Damilare Adedoyin: [C:03+2] Remove GravyPaymentResponseFactory::fromRawResponse() [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1047157 (https://phabricator.wikimedia.org/T364506) (owner: 10Jgleeson) [15:27:35] (03CR) 10Damilare Adedoyin: [C:03+2] Extract out Gravy amount transformation to function [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1047159 (https://phabricator.wikimedia.org/T364506) (owner: 10Jgleeson) [15:28:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 33 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:28:29] (03CR) 10Damilare Adedoyin: [C:03+2] Minor casing fix to Gravy API param. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1047179 (https://phabricator.wikimedia.org/T367112) (owner: 10Jgleeson) [15:33:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 36 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:33:43] anilk: first glance at those fields katie wants & it seems like more of a project than I expected - I thought she wanted existing calculated data pushed up to Acoustic - but it looks like she wants us to calculate more stuff, that probably needs scoping. But what we can do is add what we have in that window & add place holders for anything we don't such that we can add it later without requiring any re-mapping [15:38:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 33 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:43:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:46:28] (03Merged) 10jenkins-bot: Auto-cancel PayPal subscriptions on chargeback [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1038400 (https://phabricator.wikimedia.org/T361243) (owner: 10Ejegg) [15:46:29] (03Merged) 10jenkins-bot: Remove GravyPaymentResponseFactory::fromRawResponse() [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1047157 (https://phabricator.wikimedia.org/T364506) (owner: 10Jgleeson) [15:46:30] (03Merged) 10jenkins-bot: Extract out Gravy amount transformation to function [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1047159 (https://phabricator.wikimedia.org/T364506) (owner: 10Jgleeson) [15:48:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:53:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 33 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [15:58:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 36 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:02:09] thanks for the review damilare ! [16:03:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:08:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:13:13] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:18:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 32 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:22:17] inp [16:23:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:28:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:28:59] thanks eileen - I can mention on the phab [16:29:59] oh eileen I see you did [16:33:19] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 35 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:38:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 33 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:43:13] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 33 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:48:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 37 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:53:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 33 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [16:58:13] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 36 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [17:03:13] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 36 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [17:08:13] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 33 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [17:13:15] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [17:17:21] ACKNOWLEDGEMENT - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Ingenico:9) 9 [=1], ipset_error 34 [=1] Dwisehaupt debugging package changes that affected these checks. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [18:47:59] 06Fundraising-Backlog, 10FR-donorservices: Endowment 'Thank You' email/receipt sent in English to non-English donors - https://phabricator.wikimedia.org/T367594#9907732 (10SHust) That is what I hoped to see but after reviewing the CIDs, they are each from their respective countries! CID 64125014 Referrer: do... [18:53:15] RECOVERY - check_log_messages on frav1003 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages