[00:46:46] 10Fundraising-Backlog, 10fundraising_sprint_l_2023: Data Axle project - List pull from Civi - https://phabricator.wikimedia.org/T336891 (10AnnWF) [00:49:41] 10Fundraising-Backlog, 10fundraising_sprint_l_2023: Data Axle project - List pull from Civi - https://phabricator.wikimedia.org/T336891 (10AnnWF) [00:58:30] 10Fundraising-Backlog, 10fundraising_sprint_l_2023: Data Axle project - List pull from Civi - https://phabricator.wikimedia.org/T336891 (10AnnWF) [02:04:52] 10Fundraising Sprint Keep ya HEAD(1) up, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops: Update WMF_donor fields - https://phabricator.wikimedia.org/T331919 (10Eileenmcnaughton) I have created a spreadsheet of what fields we are looking at https://docs.google.com/spreadshee... [02:49:29] (03PS2) 10Eileen: Restructure CalculateData class such that the sql for fields is in the array declaration. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/927321 (https://phabricator.wikimedia.org/T336305) [02:49:31] (03PS7) 10Eileen: Add WMF Donor api & test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/925201 (https://phabricator.wikimedia.org/T336305) [02:49:33] (03PS2) 10Eileen: Add handling to support only getting some fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/927592 (https://phabricator.wikimedia.org/T336305) [03:23:20] (03PS3) 10Eileen: Add handling to support only getting some fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/927592 (https://phabricator.wikimedia.org/T336305) [04:18:54] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising_sprint_l_2023, 10fr-donorservices: Wrap merges in a transaction - https://phabricator.wikimedia.org/T335382 (10Eileenmcnaughton) Just noting the status here is possibly that we need to decline this phab as not being the answer & go back... [04:20:25] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising_sprint_l_2023, 10Fr-drupal-upgrade-2021: Determine maturity of Civi-standalone - https://phabricator.wikimedia.org/T337337 (10Eileenmcnaughton) Perhaps this is something I could do in San Francisco when I catch up with the core team - n... [04:22:58] 10Fundraising-Backlog: Civi Export: What to do with the MatchingGifts file and mystery fields - https://phabricator.wikimedia.org/T338259 (10Eileenmcnaughton) Note that this is the link to our docs on it https://wikitech.wikimedia.org/wiki/Fundraising/Data_and_Integrated_Processes/Acoustic_Integration#Matching_G... [05:20:29] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising_sprint_l_2023: WMF donor fields are running out - https://phabricator.wikimedia.org/T322176 (10greg) [05:44:36] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_38) - 10https://gerrit.wikimedia.org/r/927841 (owner: 10L10n-bot) [05:59:36] (03PS4) 10Eileen: Add handling to support only getting some fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/927592 (https://phabricator.wikimedia.org/T336305) [05:59:38] (03PS1) 10Eileen: Add calculation for segment fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/927966 (https://phabricator.wikimedia.org/T336305) [06:07:39] (03PS5) 10Eileen: Add handling to support only getting some fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/927592 (https://phabricator.wikimedia.org/T336305) [06:08:29] (03PS2) 10Eileen: Add calculation for segment fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/927966 (https://phabricator.wikimedia.org/T336305) [06:09:21] (03PS3) 10Eileen: Add calculation for segment fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/927966 (https://phabricator.wikimedia.org/T336305) [06:12:47] (03CR) 10CI reject: [V: 04-1] Add calculation for segment fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/927966 (https://phabricator.wikimedia.org/T336305) (owner: 10Eileen) [06:21:18] PROBLEM - check_puppetrun on pay-lvs1004 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 5 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1004&service=check_puppetrun [06:21:22] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 5 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [06:22:18] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 6 minutes ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [06:23:18] PROBLEM - check_puppetrun on frpig1001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 7 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1001&service=check_puppetrun [06:26:18] PROBLEM - check_puppetrun on pay-lvs1004 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 10 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1004&service=check_puppetrun [06:26:22] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 10 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [06:27:16] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 11 minutes ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [06:28:18] PROBLEM - check_puppetrun on frpig1001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 12 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1001&service=check_puppetrun [06:31:18] PROBLEM - check_puppetrun on pay-lvs1004 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 8 seconds ago with 33 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1004&service=check_puppetrun [06:31:22] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 47 seconds ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [06:32:18] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 1 minute ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [06:33:18] PROBLEM - check_puppetrun on frpig1001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 2 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1001&service=check_puppetrun [06:36:18] PROBLEM - check_puppetrun on pay-lvs1004 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 5 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1004&service=check_puppetrun [06:36:22] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 5 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [06:37:18] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 6 minutes ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [06:38:18] RECOVERY - check_puppetrun on frpig1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1001&service=check_puppetrun [06:41:18] PROBLEM - check_puppetrun on pay-lvs1004 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 1 minute ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1004&service=check_puppetrun [06:41:22] RECOVERY - check_puppetrun on frpig2001 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [06:42:18] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 1 minute ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [06:46:18] PROBLEM - check_puppetrun on pay-lvs1004 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 50 seconds ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1004&service=check_puppetrun [06:47:18] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 50 seconds ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [06:47:25] (03PS4) 10Eileen: Add calculation for segment fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/927966 (https://phabricator.wikimedia.org/T336305) [06:51:12] PROBLEM - check_puppetrun on frbast1001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 5 minutes ago with 30 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast1001&service=check_puppetrun [06:51:12] PROBLEM - check_puppetrun on pay-lvs2001 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 5 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [06:51:12] PROBLEM - check_puppetrun on pay-lvs2002 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 5 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [06:51:18] PROBLEM - check_puppetrun on pay-lvs1004 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 5 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1004&service=check_puppetrun [06:51:22] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 6 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [06:52:16] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 1 minute ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [06:53:18] PROBLEM - check_puppetrun on frpig1001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 7 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1001&service=check_puppetrun [06:56:12] PROBLEM - check_puppetrun on frbast1001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 10 minutes ago with 30 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast1001&service=check_puppetrun [06:56:12] PROBLEM - check_puppetrun on pay-lvs2001 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 10 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [06:56:12] PROBLEM - check_puppetrun on pay-lvs2002 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 10 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [06:56:18] PROBLEM - check_puppetrun on pay-lvs1004 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 10 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1004&service=check_puppetrun [06:56:22] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 11 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [06:57:18] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 6 minutes ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [06:58:12] PROBLEM - check_puppetrun on frpig1001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 12 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1001&service=check_puppetrun [07:01:12] PROBLEM - check_puppetrun on frbast1001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 24 seconds ago with 30 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast1001&service=check_puppetrun [07:01:12] PROBLEM - check_puppetrun on pay-lvs2001 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 16 seconds ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [07:01:14] PROBLEM - check_puppetrun on pay-lvs2002 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 25 seconds ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [07:01:18] PROBLEM - check_puppetrun on pay-lvs1004 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 7 seconds ago with 33 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1004&service=check_puppetrun [07:01:22] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 26 seconds ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [07:02:18] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 58 seconds ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [07:03:18] PROBLEM - check_puppetrun on frpig1001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 2 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1001&service=check_puppetrun [07:03:57] (03PS5) 10Eileen: Add calculation for segment fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/927966 (https://phabricator.wikimedia.org/T336305) [07:05:02] ACKNOWLEDGEMENT - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 58 seconds ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] Dwisehaupt Something is up with puppet runs on the buster hosts. they are falling back to an empty server setting and trying to contact localh [07:05:02] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [07:05:02] ACKNOWLEDGEMENT - check_puppetrun on frbast1001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 24 seconds ago with 30 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] Dwisehaupt Something is up with puppet runs on the buster hosts. they are falling back to an empty server setting and trying to contact loca [07:05:02] 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast1001&service=check_puppetrun [07:05:02] ACKNOWLEDGEMENT - check_puppetrun on frpig1001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 2 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] Dwisehaupt Something is up with puppet runs on the buster hosts. they are falling back to an empty server setting and trying to contact localh [07:05:02] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1001&service=check_puppetrun [07:05:03] ACKNOWLEDGEMENT - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 26 seconds ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] Dwisehaupt Something is up with puppet runs on the buster hosts. they are falling back to an empty server setting and trying to contact local [07:05:03] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [07:05:03] ACKNOWLEDGEMENT - check_puppetrun on pay-lvs1004 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 7 seconds ago with 33 failures. Failed resources (up to 3 shown): Dwisehaupt Something is up with puppet runs on the buster hosts. they are falling back to an empty server setting and trying to contact localhost:8140 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1004&service=check_puppetrun [07:05:04] ACKNOWLEDGEMENT - check_puppetrun on pay-lvs2001 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 16 seconds ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] Dwisehaupt Something is up with puppet runs on the buster hosts. they are falling back to an empty server setting and trying to contact loc [07:05:04] 40 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [07:05:05] ACKNOWLEDGEMENT - check_puppetrun on pay-lvs2002 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 25 seconds ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] Dwisehaupt Something is up with puppet runs on the buster hosts. they are falling back to an empty server setting and trying to contact loc [07:06:12] RECOVERY - check_puppetrun on pay-lvs2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [07:06:50] ugh. looks like those errors may be related to a libruby upgrade [07:07:12] PROBLEM - check_puppetrun on pay-lvs1003 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 6 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1003&service=check_puppetrun [07:07:12] PROBLEM - check_puppetrun on frav1002 is CRITICAL: CRITICAL: Puppet has 45 failures. Last run 6 minutes ago with 45 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_puppetrun [07:07:18] PROBLEM - check_puppetrun on frban1001 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 6 minutes ago with 32 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_puppetrun [07:08:18] PROBLEM - check_puppetrun on frbast2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 7 minutes ago with 30 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2001&service=check_puppetrun [07:08:18] RECOVERY - check_puppetrun on frpig1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1001&service=check_puppetrun [07:10:12] PROBLEM - check_puppetrun on frauth2001 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 9 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2001&service=check_puppetrun [07:10:18] PROBLEM - check_puppetrun on frdev1001 is CRITICAL: CRITICAL: Puppet has 41 failures. Last run 9 minutes ago with 41 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_puppetrun [07:11:12] RECOVERY - check_puppetrun on frbast1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast1001&service=check_puppetrun [07:11:18] RECOVERY - check_puppetrun on pay-lvs2002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [07:11:22] RECOVERY - check_puppetrun on frpig2001 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [07:12:12] PROBLEM - check_puppetrun on pay-lvs1003 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 11 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1003&service=check_puppetrun [07:12:18] PROBLEM - check_puppetrun on frav1002 is CRITICAL: CRITICAL: Puppet has 45 failures. Last run 11 minutes ago with 45 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_puppetrun [07:12:18] PROBLEM - check_puppetrun on frban1001 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 11 minutes ago with 32 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_puppetrun [07:13:18] PROBLEM - check_puppetrun on frbast2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 12 minutes ago with 30 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2001&service=check_puppetrun [07:15:12] PROBLEM - check_puppetrun on frauth2001 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 14 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2001&service=check_puppetrun [07:15:18] PROBLEM - check_puppetrun on frdev1001 is CRITICAL: CRITICAL: Puppet has 41 failures. Last run 14 minutes ago with 41 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_puppetrun [07:16:12] PROBLEM - check_puppetrun on pay-lvs2001 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 49 seconds ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [07:17:12] PROBLEM - check_puppetrun on frav1002 is CRITICAL: CRITICAL: Puppet has 45 failures. Last run 1 minute ago with 45 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_puppetrun [07:17:12] PROBLEM - check_puppetrun on pay-lvs1003 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 1 minute ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1003&service=check_puppetrun [07:17:18] PROBLEM - check_puppetrun on frban1001 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 49 seconds ago with 32 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_puppetrun [07:18:12] PROBLEM - check_puppetrun on frbast2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 2 minutes ago with 30 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2001&service=check_puppetrun [07:20:12] PROBLEM - check_puppetrun on frauth2001 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 4 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2001&service=check_puppetrun [07:20:18] RECOVERY - check_puppetrun on frdev1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_puppetrun [07:21:12] PROBLEM - check_puppetrun on pay-lvs2001 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 5 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [07:21:18] PROBLEM - check_puppetrun on pay-lvs2002 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 5 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [07:21:22] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 5 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [07:22:12] RECOVERY - check_puppetrun on pay-lvs1003 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1003&service=check_puppetrun [07:22:12] PROBLEM - check_puppetrun on frav1002 is CRITICAL: CRITICAL: Puppet has 45 failures. Last run 6 minutes ago with 45 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_puppetrun [07:22:18] ACKNOWLEDGEMENT - check_puppetrun on frauth2001 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 4 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] Dwisehaupt still tracking the ruby/libruby upgrade that hosed this. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fraut [07:22:18] vice=check_puppetrun [07:22:18] ACKNOWLEDGEMENT - check_puppetrun on frav1002 is CRITICAL: CRITICAL: Puppet has 45 failures. Last run 6 minutes ago with 45 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] Dwisehaupt still tracking the ruby/libruby upgrade that hosed this. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav100 [07:22:18] =check_puppetrun [07:22:19] ACKNOWLEDGEMENT - check_puppetrun on frban1001 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 49 seconds ago with 32 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] Dwisehaupt still tracking the ruby/libruby upgrade that hosed this. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban [07:22:19] ice=check_puppetrun [07:22:19] ACKNOWLEDGEMENT - check_puppetrun on frbast2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 2 minutes ago with 30 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] Dwisehaupt still tracking the ruby/libruby upgrade that hosed this. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbas [07:22:20] vice=check_puppetrun [07:22:20] ACKNOWLEDGEMENT - check_puppetrun on pay-lvs2001 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 5 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] Dwisehaupt still tracking the ruby/libruby upgrade that hosed this. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay- [07:22:21] ervice=check_puppetrun [07:23:12] RECOVERY - check_puppetrun on frbast2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2001&service=check_puppetrun [07:23:18] PROBLEM - check_puppetrun on frpig1001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 6 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1001&service=check_puppetrun [07:26:12] PROBLEM - check_puppetrun on frbast1001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 9 minutes ago with 30 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast1001&service=check_puppetrun [07:26:18] PROBLEM - check_puppetrun on pay-lvs2002 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 10 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [07:26:22] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 10 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [07:28:18] PROBLEM - check_puppetrun on frpig1001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 11 minutes ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1001&service=check_puppetrun [07:30:12] RECOVERY - check_puppetrun on frauth2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2001&service=check_puppetrun [07:31:12] PROBLEM - check_puppetrun on frbast1001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 21 seconds ago with 30 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast1001&service=check_puppetrun [07:31:14] PROBLEM - check_puppetrun on pay-lvs2002 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 1 minute ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [07:31:22] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 17 seconds ago with 36 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [07:36:12] RECOVERY - check_puppetrun on pay-lvs2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [07:37:12] PROBLEM - check_puppetrun on pay-lvs1003 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 6 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1003&service=check_puppetrun [07:38:18] PROBLEM - check_puppetrun on frbast2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 6 minutes ago with 30 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2001&service=check_puppetrun [07:38:18] RECOVERY - check_puppetrun on frpig1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1001&service=check_puppetrun [07:40:12] PROBLEM - check_puppetrun on frauth2001 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 8 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2001&service=check_puppetrun [07:40:18] PROBLEM - check_puppetrun on frdev1001 is CRITICAL: CRITICAL: Puppet has 41 failures. Last run 9 minutes ago with 41 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_puppetrun [07:40:46] (03PS6) 10Eileen: Add calculation for segment fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/927966 (https://phabricator.wikimedia.org/T336305) [07:41:10] RECOVERY - check_puppetrun on frbast1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast1001&service=check_puppetrun [07:41:16] RECOVERY - check_puppetrun on pay-lvs2002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [07:41:22] RECOVERY - check_puppetrun on frpig2001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [07:42:12] PROBLEM - check_puppetrun on pay-lvs1003 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 11 minutes ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1003&service=check_puppetrun [07:43:18] PROBLEM - check_puppetrun on frbast2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 12 minutes ago with 30 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2001&service=check_puppetrun [07:45:12] PROBLEM - check_puppetrun on frauth2001 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 13 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2001&service=check_puppetrun [07:45:18] PROBLEM - check_puppetrun on frdev1001 is CRITICAL: CRITICAL: Puppet has 41 failures. Last run 14 minutes ago with 41 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_puppetrun [07:47:12] PROBLEM - check_puppetrun on pay-lvs1003 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 1 minute ago with 33 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1003&service=check_puppetrun [07:47:12] RECOVERY - check_puppetrun on frav1002 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_puppetrun [07:48:12] PROBLEM - check_puppetrun on frbast2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 2 minutes ago with 30 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2001&service=check_puppetrun [07:50:12] RECOVERY - check_puppetrun on frauth2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2001&service=check_puppetrun [07:50:18] RECOVERY - check_puppetrun on frdev1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_puppetrun [07:52:19] RECOVERY - check_puppetrun on pay-lvs1003 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1003&service=check_puppetrun [07:52:20] RECOVERY - check_puppetrun on civi2001 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [07:52:20] RECOVERY - check_puppetrun on frban1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_puppetrun [07:53:15] RECOVERY - check_puppetrun on frbast2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2001&service=check_puppetrun [07:56:19] RECOVERY - check_puppetrun on pay-lvs1004 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1004&service=check_puppetrun [08:03:22] ^^ ok. this should be fixed. wrap up email being sent. [08:10:19] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 6 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [08:15:15] RECOVERY - check_puppetrun on frdata1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [08:40:19] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 6 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [08:45:17] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 11 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [08:50:15] RECOVERY - check_puppetrun on frdata1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [09:10:19] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 6 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [09:15:19] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 11 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [09:20:17] RECOVERY - check_puppetrun on frdata1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [09:40:19] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 6 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [09:45:19] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 11 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [09:50:19] RECOVERY - check_puppetrun on frdata1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [09:57:13] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 6 minutes ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [10:02:15] RECOVERY - check_puppetrun on civi2001 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [10:10:15] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 6 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [10:15:15] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 11 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [10:20:15] RECOVERY - check_puppetrun on frdata1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [10:27:15] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 6 minutes ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [10:32:17] RECOVERY - check_puppetrun on civi2001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [10:40:17] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 6 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [10:45:17] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 11 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [10:50:17] RECOVERY - check_puppetrun on frdata1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [10:57:15] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 6 minutes ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [11:02:17] RECOVERY - check_puppetrun on civi2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [11:10:17] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 6 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [11:15:17] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 11 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [11:17:09] 10Fundraising-Backlog: Pre-notification received although the recurring set up is canceled in Civi - India - https://phabricator.wikimedia.org/T338309 (10RKumar_WMF) [11:20:17] RECOVERY - check_puppetrun on frdata1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [11:27:17] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 6 minutes ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [11:32:15] RECOVERY - check_puppetrun on civi2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [11:33:38] (03CR) 10Nikerabbit: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_38) - 10https://gerrit.wikimedia.org/r/927841 (owner: 10L10n-bot) [11:40:21] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 6 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [11:45:15] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 11 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [11:50:15] RECOVERY - check_puppetrun on frdata1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [11:52:15] PROBLEM - check_puppetrun on frban1001 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 5 minutes ago with 32 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_puppetrun [11:57:15] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 6 minutes ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [11:57:15] PROBLEM - check_puppetrun on frban1001 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 10 minutes ago with 32 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_puppetrun [12:02:18] RECOVERY - check_puppetrun on civi2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [12:02:18] RECOVERY - check_puppetrun on frban1001 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_puppetrun [12:10:18] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 6 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [12:15:18] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 11 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [12:20:18] RECOVERY - check_puppetrun on frdata1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [12:27:18] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 6 minutes ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [12:32:18] RECOVERY - check_puppetrun on civi2001 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [12:40:18] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 6 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [12:45:18] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 11 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [12:50:18] RECOVERY - check_puppetrun on frdata1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [12:57:18] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 6 minutes ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [13:02:18] RECOVERY - check_puppetrun on civi2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [13:10:18] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 6 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [13:15:18] RECOVERY - check_puppetrun on frdata1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [13:27:18] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 6 minutes ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [13:29:20] (03PS6) 10Damilare Adedoyin: Create recurring upgrade form [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/927178 (https://phabricator.wikimedia.org/T335795) [13:32:15] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 57 seconds ago with 37 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [13:37:15] RECOVERY - check_puppetrun on civi2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [13:40:21] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 6 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [13:45:15] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 11 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [13:47:40] (03PS7) 10Damilare Adedoyin: Create recurring upgrade form [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/927178 (https://phabricator.wikimedia.org/T335795) [13:50:15] RECOVERY - check_puppetrun on frdata1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [14:10:18] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 6 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [14:15:18] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 11 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [14:20:14] RECOVERY - check_puppetrun on frdata1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [14:24:19] (03PS1) 10Damilare Adedoyin: Handle submission of new recurring upgrade form [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/928066 (https://phabricator.wikimedia.org/T335796) [14:25:59] (03CR) 10CI reject: [V: 04-1] Handle submission of new recurring upgrade form [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/928066 (https://phabricator.wikimedia.org/T335796) (owner: 10Damilare Adedoyin) [14:58:13] (03PS8) 10Damilare Adedoyin: Create recurring upgrade form [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/927178 (https://phabricator.wikimedia.org/T335795) [14:59:53] (03CR) 10CI reject: [V: 04-1] Create recurring upgrade form [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/927178 (https://phabricator.wikimedia.org/T335795) (owner: 10Damilare Adedoyin) [15:00:50] (03PS9) 10Damilare Adedoyin: Create recurring upgrade form [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/927178 (https://phabricator.wikimedia.org/T335795) [15:04:39] (03PS2) 10Damilare Adedoyin: Handle submission of new recurring upgrade form [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/928066 (https://phabricator.wikimedia.org/T335796) [15:06:35] (03CR) 10CI reject: [V: 04-1] Handle submission of new recurring upgrade form [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/928066 (https://phabricator.wikimedia.org/T335796) (owner: 10Damilare Adedoyin) [15:09:01] (03PS3) 10Damilare Adedoyin: Handle submission of new recurring upgrade form [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/928066 (https://phabricator.wikimedia.org/T335796) [15:10:40] 10Fundraising Sprint Last Christmas I gave you my donation, 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 10Recurring-Donations, and 2 others: Handle submission of new recurring upgrade form - https://phabricator.wikimedia.org/T335796 (10Damilare) a:03Damilare [15:12:58] (03PS4) 10Damilare Adedoyin: Handle submission of new recurring upgrade form [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/928066 (https://phabricator.wikimedia.org/T335796) [15:16:15] wfan I see you are currently reviewing the recurring upgrade page patch, I made some changes to the patch today so you may need to pull in the latest. Thanks! [15:17:15] damilare: I don't see wfan here right now *shrug* [15:18:33] ohh thanks greg-g. [15:19:21] (wfan didn't turn into a color in my client, which is how I noticed) [15:22:18] PROBLEM - check_puppetrun on frban1001 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 5 minutes ago with 32 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_puppetrun [15:24:57] yep I'm using the terminal client greg-g, I should probably add that feature if it exists [15:27:22] PROBLEM - check_puppetrun on frban1001 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 10 minutes ago with 32 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_puppetrun [15:32:18] RECOVERY - check_puppetrun on frban1001 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_puppetrun [16:00:15] damilare: a few years ago I finally switched from irssi+ssh (well, mosh) to weechat+glowing-bear.org as the front-end [16:09:32] greg-g: I have weechat but haven't quite made the leap to glowing bear. a few years back, Casey who was our ops guy, kindly wrote a guide for setting it up but I think it's still on my to-read list :| [16:18:22] 10Fundraising-Backlog, 10MediaWiki-extensions-FundraiserLandingPage, 10donate.wikimedia.org: donate.wikimedia.org landing pages don't have descriptions in search engine results - https://phabricator.wikimedia.org/T254808 (10AKanji-WMF) Moving back to Triage so we can re-evaluate as a team [16:18:33] 10Fundraising Sprint Keep ya HEAD(1) up, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops: Update WMF_donor fields - https://phabricator.wikimedia.org/T331919 (10ERoden-WMF) Thanks @AKanji-WMF & @Eileenmcnaughton. I've just requested access to the spreadsheet and will add some... [16:33:24] 10Fundraising Sprint Last Christmas I gave you my donation, 10Fundraising-Backlog: Data Axle project - List pull from Civi - https://phabricator.wikimedia.org/T336891 (10AnnWF) [16:34:23] 10fundraising-tech-ops, 10FR-Email: Update Acoustic Unsubscribe landing page domain to be served by a secure wikimedia.org domain - https://phabricator.wikimedia.org/T336000 (10EWilfong_WMF) Just noting that Acoustic has said they plan to update this process to follow the one used for link tracking (links.emai... [17:31:24] jgleeson: greg-g I had glowing bear setup, cut it out for a bit to save costs. I'd probably bring it back up soon. [17:35:30] 10Fundraising-Backlog, 10FR-dlocal, 10Recurring-Donations: dLocal recurring INR success rates for 2nd month, post on-demand switch - https://phabricator.wikimedia.org/T338351 (10MBeat33) [17:36:55] 10Fundraising-Backlog, 10FR-dlocal, 10Recurring-Donations: dLocal recurring INR success rates for 2nd month, post on-demand switch - https://phabricator.wikimedia.org/T338351 (10MBeat33) [17:38:29] 10Fundraising-Backlog, 10FR-dlocal, 10Recurring-Donations: dLocal recurring INR success rates for 2nd month, post on-demand switch - https://phabricator.wikimedia.org/T338351 (10EMartin) I also ran this report out of Dlocal and find that most of the IR transactions are rejecting over the month: {F37097034} [17:55:05] Hi frtech, we are seeing most/all of our India recurring donations rejecting https://phabricator.wikimedia.org/T338351. I have reached out to Dlocal with these findings as well. I will loop you in to the thread with them. It seems to be some sort of 'acquirer reject' but we don't yet know root cause. [17:58:39] thanks emartin_ I think that was the same error we saw in testing with Rakhi, the Error in Acquirer [18:03:16] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Adyen, 10fr-donorservices: question re: Adyen audit file May 25th - https://phabricator.wikimedia.org/T337881 (10Cstone) [18:22:01] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Adyen, 10fr-donorservices: question re: Adyen audit file May 25th - https://phabricator.wikimedia.org/T337881 (10Cstone) Hi @MBeat33 something odd happened here. I'll get it into civi manually so we can get them a TY email and then look into the... [18:23:18] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Adyen, 10fr-donorservices: question re: Adyen audit file May 25th - https://phabricator.wikimedia.org/T337881 (10MBeat33) Thank you, @Cstone [19:00:22] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Adyen, 10fr-donorservices: question re: Adyen audit file May 25th - https://phabricator.wikimedia.org/T337881 (10Cstone) I created them in civi: cid=59717237 I manually added them and the donation so the TY email didn't autosend. [19:15:21] 10Fundraising-Backlog: make continue button (not credit card selection) default on cc forms - https://phabricator.wikimedia.org/T237761 (10AKanji-WMF) 05Open→03Resolved a:03AKanji-WMF Closing as per convo with @XenoRyet. [19:25:12] 10Fundraising Sprint Last Christmas I gave you my donation, 10Fundraising-Backlog: Goal - Sprint L 2 - Handoff - https://phabricator.wikimedia.org/T338271 (10AKanji-WMF) a:03Ejegg [19:40:57] 10Fundraising-Backlog: Civi Export: What to do with the MatchingGifts file and mystery fields - https://phabricator.wikimedia.org/T338259 (10KHaggard) Ok, thanks @Eileenmcnaughton . I just read the documentation and confirming on my side that those last two columns in the chart don't exist in Acoustic. We never... [19:43:04] 10Fundraising Sprint Last Christmas I gave you my donation, 10Fundraising-Backlog: Data Axle project - List pull from Civi - https://phabricator.wikimedia.org/T336891 (10AnnWF) [19:43:39] 10Fundraising Sprint Last Christmas I gave you my donation, 10Fundraising-Backlog: Data Axle project - List pull from Civi - https://phabricator.wikimedia.org/T336891 (10AnnWF) [20:04:58] 10Fundraising-Backlog: Rotate the maxmind minfraud api key - https://phabricator.wikimedia.org/T338382 (10Dwisehaupt) [20:22:32] 10Fundraising Sprint Keep ya HEAD(1) up, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops: Update WMF_donor fields - https://phabricator.wikimedia.org/T331919 (10Eileenmcnaughton) I updated sharing settings to the foundation [20:32:26] 10Fundraising-Backlog: Civi Export: What to do with the MatchingGifts file and mystery fields - https://phabricator.wikimedia.org/T338259 (10KHaggard) Hi all, just following up. @NNichols and I chat over slack and here is more context from Nora: "Sorry I am having some login issues with phab. The fields you're... [21:12:15] (03PS1) 10Eileen: Fix thank-you regression bugs [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/928142 (https://phabricator.wikimedia.org/T309348) [21:12:29] cstone: I just found a regression - are you able to review ^^ with some urgency? [21:12:45] sure [21:13:04] I can hang out if you want [21:13:07] ok [21:13:17] give e 5 [21:14:16] eileen: i think that phab linked there may not be the right one? [21:17:53] no perhaps not [21:17:55] .... [21:18:50] (03PS2) 10Eileen: Fix thank-you regression bugs [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/928142 (https://phabricator.wikimedia.org/T309349) [21:19:12] cstone: linked now to ^^ [21:19:44] it is kinda general topic area rather than the exact thing cos it is a regression from fixes related [21:20:06] want to go back in standup eileen ? [21:20:11] sure [21:34:14] (03CR) 10Cstone: [C: 03+2] "Thanks for helping with the demo!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/928142 (https://phabricator.wikimedia.org/T309349) (owner: 10Eileen) [21:48:19] (03Merged) 10jenkins-bot: Fix thank-you regression bugs [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/928142 (https://phabricator.wikimedia.org/T309349) (owner: 10Eileen) [21:48:57] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/928145 [21:48:59] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/928166 [21:49:50] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/928166 (owner: 10Eileen) [21:57:49] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/928167 [21:57:57] (03Abandoned) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/928145 (owner: 10Eileen) [21:58:10] (03Abandoned) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/928166 (owner: 10Eileen) [21:58:48] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/928167 (owner: 10Eileen) [21:59:37] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/928167 (owner: 10Eileen) [22:05:39] !log civicrm upgraded from bcc8fccc to 066095b8 [22:05:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:31:29] looks like anilk has left - XenoRyet greg-g do you know where we are at with the core team search kit work - just worried about Coleman not having enough time [22:35:00] 10Fundraising-Backlog, 10FR-dlocal, 10Recurring-Donations: dLocal recurring INR success rates for 2nd month, post on-demand switch - https://phabricator.wikimedia.org/T338351 (10EMartin) From Dlocal: Hi Evelyn. I checked some of the transactions. Transactions with error code 321 are related to subscriptio... [22:46:55] I think Greg is in process of getting it through Coupa [22:50:47] XenoRyet: ok cool - did you agree the scope with Coleman? I did chat with him & I think for the $ we had we taked about adding saved_search_id to action schedule & making it work from that & some protections against in dvertant edits - but still requiring dev config rather than a full UI [22:52:03] also if it is definitely happening & just a matter of admin he will start on it (in the past he has had to do that to get the work done between when we send the contract & when we extpect it done by...) [22:52:17] PROBLEM - check_puppetrun on frban1001 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 5 minutes ago with 32 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_puppetrun [22:52:54] I'd leave that question for Greg. [22:57:19] RECOVERY - check_puppetrun on frban1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_puppetrun [23:08:24] eileen: definitely happening, contract approval is going through now [23:08:36] greg-g: I'll let him know [23:08:52] thumbs up [23:34:05] (03PS1) 10Eileen: Thank you preview - do not assign trxn_id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/928172 (https://phabricator.wikimedia.org/T309349) [23:35:58] cstone: another minor one with the preview ^^ - only affects the preview