[00:00:02] thanks wfan [00:00:07] i'll take a look in a bit [00:00:18] Thanks :) [00:10:47] !log updated payments-wiki from 53a7b7bd to 2f95d8b4 [00:10:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:59:57] (03CR) 10Ejegg: [C: 03+2] Implement separate auth and capture steps for Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/812016 (https://phabricator.wikimedia.org/T312289) (owner: 10Damilare Adedoyin) [02:00:30] (03Merged) 10jenkins-bot: Implement separate auth and capture steps for Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/812016 (https://phabricator.wikimedia.org/T312289) (owner: 10Damilare Adedoyin) [02:31:28] (03CR) 10Ejegg: "This is looking pretty good, and is an impressive amount of work!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/812100 (https://phabricator.wikimedia.org/T310757) (owner: 10Wfan) [02:52:10] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [02:57:10] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [03:05:14] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Ingenico audit parser sending already-refunded donations to refund queue - https://phabricator.wikimedia.org/T312824 (10Ejegg) [03:12:43] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Ingenico audit parser sending already-refunded donations to refund queue - https://phabricator.wikimedia.org/T312824 (10Ejegg) Pre-prod key has been regenerated and updated in the SmashPig staging yaml file up on the puppetmaster. I hav... [03:15:33] 10Fundraising Sprint Machine Unlearning, 10Fundraising-Backlog: Ingenico/Worldline production key expiring 14 July - https://phabricator.wikimedia.org/T312662 (10Ejegg) Pre-prod key has been regenerated and updated in the SmashPig staging yaml file up on the puppetmaster. I haven't done the production or sandb... [05:17:22] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/FundraisingEmailUnsubscribe] (REL1_37) - 10https://gerrit.wikimedia.org/r/813035 (owner: 10L10n-bot) [05:17:30] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_37) - 10https://gerrit.wikimedia.org/r/813029 (owner: 10L10n-bot) [07:52:27] (03CR) 10Abijeet Patro: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_37) - 10https://gerrit.wikimedia.org/r/813029 (owner: 10L10n-bot) [07:54:07] (03CR) 10Abijeet Patro: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/FundraisingEmailUnsubscribe] (REL1_37) - 10https://gerrit.wikimedia.org/r/813035 (owner: 10L10n-bot) [08:00:27] (03PS1) 10Samwilson: Fix syntax error in email_forms/i18n/qqq.json and add to CI [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/813191 [14:07:00] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10krobinson) Reopening this one as we are seeing many more of these come through since launching yesterday. Handful of examples: ZD 1136675 / CID 56104535 (spain / card) ZD 11366... [14:29:40] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10TSkaff) It seems like some of them are seeing the "session expired, please try again" error message? krobinson have we ever see this in other countries where we've tried MC like... [14:58:55] howdy fr-tech [15:04:42] hellosie-ohs jgleeson fr-tech :) [15:04:51] how is everyone doing today jgleeson? [15:06:06] hi jgleeson AndyRussG fr-tech [15:08:11] hi jgleeson AndyRussG cstone fr-tech [15:10:29] :) [15:10:54] howdy AndyRussG cstone damilare! [15:12:25] o/ [15:13:10] Everyone is all good thanks AndyRussG. The only minor hiccup was at Tilly's first school sports day today. A kid accidentally ran into her right at the start so she cried the entire time :\ [15:13:42] and then I locked my car keys in my car. yep [15:15:07] ohhh sorry about those mishaps jgleeson..... [15:16:17] my car has an electrical fault atm so I have to manually lock each door from the inside when getting out and today I wasn't paying attention and got out the car, pushed in the lock latch on the driver door, pushed it shut and then stood there staring at my keys still in the ignition [15:17:40] luckily there was a locksmith pretty close to where it happened and they opened it up soon enough [15:18:08] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10krobinson) @TSkaff - confirmed via ZD 1136760 that is what they are seeing. We have not seen this before from donors, in Sweden or Italy - or any recent campaign. [15:23:22] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10TSkaff) I was just digging around for historical info if it helps anyone: Italy = Post-payment MC **Ingenico** Sweden = Post-payment MC **Adyen** (but the campaign ran at 75% tr... [15:29:12] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10krobinson) If there is no other fix, is there any way we could change the message on the error page so it doesn't invite them to try again? [15:36:02] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) I have initiated an inquiry with Adyen to see if they can determine the extent of the duplicates. Because they are the same email but two separate transactions within a... [16:07:22] sorry about jgleeson, reminds of the time I drove over 10 miles only to discover I had forgotten the key fob to my keyless car in the driveway. Unfortunately for me, the realisation only came after I had turned off the car. [16:11:06] arghhhhhh damilare! yep feels like that would trigger the same feelings I had [16:19:38] yep I can imagine, luckily a friend of mine was home and brought it to me 30mins later. [16:22:45] 10Fundraising-Backlog: Name changes required for Ingenico 3S2.0 - https://phabricator.wikimedia.org/T312877 (10EMartin) [16:25:57] ahhh yeah when I had a car here years ago I locked myself out at least 3 times [16:27:29] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) So far, I have been unable to reproduce this in incognito testing and with several attempts in each bundle market. [16:29:29] I'd be updating smashpig to v0.8.0.2 for the braintree auth and capture steps [16:31:04] damilare: they feels like it might be worth a middle (minor) bump [16:31:45] if we ignore the leading 0 [16:31:45] ohh how do you mean jgleeson? [16:32:17] v0.8.1 [16:32:29] yeah! [16:34:15] cool [16:35:58] 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising Sprint Machine Unlearning, 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Braintree-Integration: Network configuration for Braintree integration - https://phabricator.wikimedia.org/T303457 (10Jgreen) Heard back from Paypal: ` 1. gsta... [16:35:59] you're right, thanks [16:45:10] damilare jgleeson fr-tech many congrats on all the braintree progress :) :) where would it be most productive for me to jump in now to help push it through? [16:54:58] sorry damilare was on a call but yeah I feel like we *try* to stay with semantic versioning although it's definitely subjective on what constitutes a major/minor/patch. We're yet to get to v1 with smashpig but hopefully that'll come soon! [16:59:59] AndyRussG: I'm not sure to be honest. I've trying wrap up the IPN stuff I've been dragging out over last week and just catching up on the status of things myself. It sounds like we're close to having production APi access? Maybe dig in on the audit review bits or it might be worth thinking out the multicurrency code changes needed for the merchant account switching, if you wanted to get in the [17:00:00] code a bit [17:03:54] I've been* [17:04:09] jgleeson: oki... yeah yesterday ej egg made a task for the multicurrency code. we decided it wasn't needed for an initial test [17:04:19] oh no that actually should have been, I'm* trying [17:04:36] https://phabricator.wikimedia.org/T312808 [17:04:45] ah ok I guess that makes sense AndyRussG as we'll be trying in USD only? [17:04:48] yeah [17:04:59] also another question is whether we need the IPN stuff for the initial test [17:05:11] I don't think we do [17:08:37] oki fanrtastic thx! [17:14:12] (03PS11) 10Wfan: Braintree Audit parse donation, refund and chargeback [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/812100 (https://phabricator.wikimedia.org/T310757) [17:19:47] (03PS12) 10Wfan: Braintree Audit parse donation, refund and chargeback [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/812100 (https://phabricator.wikimedia.org/T310757) [17:20:05] (03PS13) 10Wfan: Braintree Audit parse donation, refund and chargeback [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/812100 (https://phabricator.wikimedia.org/T310757) [17:30:59] Howdy fr-tech: I've got FR-tech office hours in a bit, and I'm sure someone is going to ask where we're at in regards to Braintree internal test. So what's the word? [17:32:14] I can see Dami's patch got merged so that just leaves us to iron out the production API credentials [17:32:28] let's see where we are with that [17:33:42] looks like an email was sent last night to Braintree asking for account access. email subject is 'Access to Braintree production console' [17:35:02] once we get that and drop 'em in, and damilare finishes his smashpig version bump, we're looking ready to me? does that sound right fr-tech? [17:35:44] oh wait a minute [17:36:40] (03PS1) 10Damilare Adedoyin: Update smashpig to v0.8.1 T303419 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/813280 [17:36:50] damilare: do we still need to finish off the second fraud step now that we have the two-stage flow [17:37:24] yeah we do jgleeson, the smashpig bump affects that too. [17:37:54] ah ok so there might be a bit more do then XenoRyet [17:38:17] to do * [17:38:47] (03PS2) 10Damilare Adedoyin: Update smashpig to v0.8.1 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/813280 [17:39:12] pushing on the prod account access would help though if we can [17:39:29] How much work is that smashpig version bump? [17:41:23] Not too much, just need to update and deploy donationInterface with the new version of smashpig in order to progress with the second fraud step [17:42:11] Good times. Do we have an idea how soon that's going to happen? [17:43:09] I should be done CR of the related DonationInterface patch in a little while, that should help? (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/811746) [17:43:58] ok I just looked over dami's patch and I think we also need to update donation interface to use the new smashpig code [17:44:23] at current, DI is not calling the newly introduced second step from what I can see. Is that right damilare ? [17:44:33] The deployment is in progress, for the fraud stuff with And AndyRussG and jgleeson input we should rounding up on the fraud patch by thursday possibly. [17:45:00] yes it is jgleeson [17:45:42] DI is currently calling the ApprovePaymentResponse on the second step I mean [17:46:06] approvePayment method* [17:46:46] i was testing the two step method but got distracted by ids again but from what i saw so far looked good [17:47:15] checkout #74 jgleeson [17:47:49] Thanks cstone [17:48:24] Could anyone please help review this patch: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/813280/ which adds the new smashpig version to DI [17:48:47] just looking at that patch now dami. I'm not familiar with the statuses Braintree moves to after the the authorize call. the code suggests it's pending [17:49:23] (03CR) 10Jgleeson: [C: 03+2] Update smashpig to v0.8.1 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/813280 (owner: 10Damilare Adedoyin) [17:52:19] (03Merged) 10jenkins-bot: Update smashpig to v0.8.1 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/813280 (owner: 10Damilare Adedoyin) [17:52:32] XenoRyet: my guess would be that the remaining bits will take us to the end of the week probably ready for testing next week [17:53:33] dependant on us getting the prod account access needed [17:54:02] Yea, I'm going to keep pressing on that front to make sure it doesn't block us. [17:54:20] fr-tech: the pending trxn resolver for braintree is quickly gonna become a priority [17:55:01] Any chance we can get the other bits by EOD Thursday so Evelyn and team can play with a bit before we go to Ireland? [17:56:24] It's actually PENDING_POKE. However I also added a check in PENDING, incase the response was set to AUTHORIZING from braintree jgleeson [17:57:22] I was just reading that damilare https://graphql.braintreepayments.com/guides/creating_transactions/#using-separate-authorization-and-capture [17:58:21] Okk [17:58:38] looks good! [17:59:35] thanks jgleeson, please do you have some time to go through this deployment process with me? [18:01:38] (03CR) 10Jgleeson: Add second round of filters (031 comment) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/811746 (https://phabricator.wikimedia.org/T311625) (owner: 10Damilare Adedoyin) [18:01:58] damilare: I'm just about to grab dinner but can do as soon as I get back [18:02:45] Cool, sure thanks [18:03:50] XenoRyet: WRT Thursday. It's possible as dami says but would be great to get the account access todayish so we can confirm no other unexpected things pop up once we switch over to prod [18:03:59] back soon [18:07:10] (03PS1) 10Damilare Adedoyin: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/813285 [18:07:20] Yea, I'm talking to Leanne right now and she's going to lean on PayPal [18:08:55] Hi AndyRussG: do you have time to review my patches, I updated based on Elliott's comments and he has a day off today, https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/SmashPig/+/808008/, https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/SmashPig/+/812100/ and https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/809644/, thanks! [18:10:20] wfan: sure! I can look a bit later today :) was just gonna prioritize stuff needed for the first test, so I guess this afternoon after tech-talk I'd look? or are they needed for the initial test, do you think? [18:10:59] Ok, I will do the review for Dami's patch today too [18:13:53] ah cool thx wfan :) :) [18:18:22] (03CR) 10Damilare Adedoyin: [C: 03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/813285 (owner: 10Damilare Adedoyin) [18:18:47] (03PS9) 10Wfan: Add second round of filters [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/811746 (https://phabricator.wikimedia.org/T311625) (owner: 10Damilare Adedoyin) [18:20:02] 10fundraising-tech-ops: Install and configure new host frdb1006 - https://phabricator.wikimedia.org/T312584 (10Jgreen) [18:20:04] 10fundraising-tech-ops: FRACK environment support for H750 RAID controller - https://phabricator.wikimedia.org/T312669 (10Jgreen) 05Open→03Resolved p:05Triage→03Medium a:03Jgreen [18:20:04] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/813285 (owner: 10Damilare Adedoyin) [18:20:06] 10fundraising-tech-ops: Install and configure new host frdb1005 - https://phabricator.wikimedia.org/T312583 (10Jgreen) [18:21:12] PROBLEM - check_puppetrun on pay-lvs2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [18:21:14] PROBLEM - check_puppetrun on pay-lvs2002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [18:21:16] PROBLEM - check_puppetrun on fran1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_puppetrun [18:21:18] PROBLEM - check_puppetrun on payments2002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_puppetrun [18:21:22] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [18:21:38] bah ^^^ [18:22:12] PROBLEM - check_puppetrun on pay-lvs1003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1003&service=check_puppetrun [18:22:14] PROBLEM - check_puppetrun on frdev1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_puppetrun [18:22:14] PROBLEM - check_puppetrun on frqueue1004 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_puppetrun [18:22:15] PROBLEM - check_raid on frbackup2002 is CRITICAL: CRITICAL: PERC 0: physical 12: UGood, physical 13: UGood https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_raid [18:22:15] PROBLEM - check_puppetrun on frpm2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpm2001&service=check_puppetrun [18:22:15] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [18:22:16] PROBLEM - check_puppetrun on frmx1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_puppetrun [18:22:16] PROBLEM - check_puppetrun on frban2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_puppetrun [18:22:16] PROBLEM - check_puppetrun on frqueue2002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_puppetrun [18:22:17] PROBLEM - check_puppetrun on frdb1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_puppetrun [18:22:18] PROBLEM - check_puppetrun on frmon2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon2001&service=check_puppetrun [18:22:20] PROBLEM - check_puppetrun on frdata2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_puppetrun [18:22:20] PROBLEM - check_puppetrun on payments1005 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_puppetrun [18:22:21] PROBLEM - check_puppetrun on frban1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_puppetrun [18:23:12] PROBLEM - check_puppetrun on frqueue2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_puppetrun [18:23:14] PROBLEM - check_puppetrun on frbast2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2001&service=check_puppetrun [18:23:16] PROBLEM - check_puppetrun on frnetmon1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frnetmon1001&service=check_puppetrun [18:23:18] PROBLEM - check_puppetrun on frqueue1003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_puppetrun [18:23:18] PROBLEM - check_puppetrun on frpig1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1001&service=check_puppetrun [18:23:19] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog1001&service=check_puppetrun [18:23:19] PROBLEM - check_puppetrun on payments2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_puppetrun [18:23:19] PROBLEM - check_puppetrun on frpm1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpm1002&service=check_puppetrun [18:23:19] PROBLEM - check_puppetrun on payments2003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_puppetrun [18:25:12] PROBLEM - check_puppetrun on frauth2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2001&service=check_puppetrun [18:25:14] PROBLEM - check_puppetrun on civi1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_puppetrun [18:25:18] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [18:25:18] PROBLEM - check_puppetrun on frdev1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_puppetrun [18:25:19] PROBLEM - check_puppetrun on frmx2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx2001&service=check_puppetrun [18:25:20] PROBLEM - check_puppetrun on payments1007 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_puppetrun [18:25:20] PROBLEM - check_puppetrun on civi1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1001&service=check_puppetrun [18:25:22] PROBLEM - check_puppetrun on payments1008 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_puppetrun [18:26:12] PROBLEM - check_puppetrun on frbast1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast1001&service=check_puppetrun [18:26:12] PROBLEM - check_puppetrun on pay-lvs2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [18:26:14] RECOVERY - check_puppetrun on pay-lvs2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [18:26:16] PROBLEM - check_puppetrun on frpm1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpm1001&service=check_puppetrun [18:26:16] RECOVERY - check_puppetrun on fran1001 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_puppetrun [18:26:18] PROBLEM - check_puppetrun on payments1006 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_puppetrun [18:26:18] PROBLEM - check_puppetrun on frmon1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon1001&service=check_puppetrun [18:26:18] PROBLEM - check_puppetrun on pay-lvs1004 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1004&service=check_puppetrun [18:26:19] PROBLEM - check_puppetrun on payments2002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_puppetrun [18:26:19] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth1001&service=check_puppetrun [18:26:20] PROBLEM - check_puppetrun on frauth1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth1002&service=check_puppetrun [18:26:22] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [18:27:12] PROBLEM - check_puppetrun on pay-lvs1003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1003&service=check_puppetrun [18:27:14] PROBLEM - check_puppetrun on frqueue1004 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_puppetrun [18:27:14] PROBLEM - check_raid on frbackup2002 is CRITICAL: CRITICAL: PERC 0: physical 12: UGood, physical 13: UGood https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_raid [18:27:15] PROBLEM - check_puppetrun on frpm2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpm2001&service=check_puppetrun [18:27:15] PROBLEM - check_puppetrun on frdev1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_puppetrun [18:27:15] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [18:27:16] PROBLEM - check_puppetrun on frmx1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_puppetrun [18:27:16] PROBLEM - check_puppetrun on frban2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_puppetrun [18:27:16] PROBLEM - check_puppetrun on frqueue2002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 11 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_puppetrun [18:27:17] PROBLEM - check_puppetrun on frdb1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_puppetrun [18:27:18] RECOVERY - check_puppetrun on frban1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_puppetrun [18:27:18] PROBLEM - check_puppetrun on frmon2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon2001&service=check_puppetrun [18:27:20] PROBLEM - check_puppetrun on payments1005 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_puppetrun [18:27:20] PROBLEM - check_puppetrun on frdata2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 11 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_puppetrun [18:28:12] PROBLEM - check_puppetrun on frqueue2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_puppetrun [18:28:12] RECOVERY - check_puppetrun on frbast2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2001&service=check_puppetrun [18:28:13] 10fundraising-tech-ops: Install and configure new host frdb1006 - https://phabricator.wikimedia.org/T312584 (10Jgreen) [18:28:14] PROBLEM - check_puppetrun on frpig1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 11 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1001&service=check_puppetrun [18:28:14] PROBLEM - check_puppetrun on frnetmon1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 11 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frnetmon1001&service=check_puppetrun [18:28:14] PROBLEM - check_puppetrun on frqueue1003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 11 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_puppetrun [18:28:15] RECOVERY - check_puppetrun on frpm1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpm1002&service=check_puppetrun [18:28:16] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 11 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog1001&service=check_puppetrun [18:28:22] PROBLEM - check_puppetrun on payments2003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_puppetrun [18:28:22] PROBLEM - check_puppetrun on payments2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 11 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_puppetrun [18:28:33] 10fundraising-tech-ops: Install and configure new host frdb1005 - https://phabricator.wikimedia.org/T312583 (10Jgreen) [18:30:12] PROBLEM - check_puppetrun on frauth2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2001&service=check_puppetrun [18:30:18] PROBLEM - check_puppetrun on civi1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_puppetrun [18:30:18] PROBLEM - check_puppetrun on frdev1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 13 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_puppetrun [18:30:18] PROBLEM - check_puppetrun on frdata1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 13 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [18:30:19] RECOVERY - check_puppetrun on payments1007 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_puppetrun [18:30:19] RECOVERY - check_puppetrun on civi1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1001&service=check_puppetrun [18:30:19] PROBLEM - check_puppetrun on frmx2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 14 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx2001&service=check_puppetrun [18:30:22] PROBLEM - check_puppetrun on payments1008 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 13 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_puppetrun [18:31:12] RECOVERY - check_puppetrun on frbast1001 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast1001&service=check_puppetrun [18:31:12] PROBLEM - check_puppetrun on pay-lvs2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 15 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [18:31:18] PROBLEM - check_puppetrun on frpm1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 13 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpm1001&service=check_puppetrun [18:31:18] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 14 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth1001&service=check_puppetrun [18:31:18] RECOVERY - check_puppetrun on frmon1001 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon1001&service=check_puppetrun [18:31:19] PROBLEM - check_puppetrun on payments1006 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 14 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_puppetrun [18:31:19] PROBLEM - check_puppetrun on pay-lvs1004 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 14 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[megacli] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1004&service=check_puppetrun [18:31:19] RECOVERY - check_puppetrun on payments2002 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_puppetrun [18:31:20] RECOVERY - check_puppetrun on frauth1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth1002&service=check_puppetrun [18:31:22] RECOVERY - check_puppetrun on frpig2001 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [18:32:12] RECOVERY - check_puppetrun on pay-lvs1003 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1003&service=check_puppetrun [18:32:12] RECOVERY - check_puppetrun on frqueue1004 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_puppetrun [18:32:13] RECOVERY - check_puppetrun on frpm2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpm2001&service=check_puppetrun [18:32:13] RECOVERY - check_puppetrun on frdev1002 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_puppetrun [18:32:14] RECOVERY - check_puppetrun on civi2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [18:32:14] PROBLEM - check_raid on frbackup2002 is CRITICAL: CRITICAL: PERC 0: physical 12: UGood, physical 13: UGood https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_raid [18:32:15] RECOVERY - check_puppetrun on frmx1001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_puppetrun [18:32:16] RECOVERY - check_puppetrun on frban2001 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_puppetrun [18:32:16] RECOVERY - check_puppetrun on frqueue2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_puppetrun [18:32:18] RECOVERY - check_puppetrun on frdb1002 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_puppetrun [18:32:18] RECOVERY - check_puppetrun on frmon2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon2001&service=check_puppetrun [18:32:19] RECOVERY - check_puppetrun on frdata2001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_puppetrun [18:32:19] RECOVERY - check_puppetrun on payments1005 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_puppetrun [18:33:12] RECOVERY - check_puppetrun on frqueue2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_puppetrun [18:33:18] RECOVERY - check_puppetrun on frnetmon1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frnetmon1001&service=check_puppetrun [18:33:18] RECOVERY - check_puppetrun on frpig1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1001&service=check_puppetrun [18:33:18] RECOVERY - check_puppetrun on frqueue1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_puppetrun [18:33:19] RECOVERY - check_puppetrun on frlog1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog1001&service=check_puppetrun [18:33:19] RECOVERY - check_puppetrun on payments2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_puppetrun [18:33:19] RECOVERY - check_puppetrun on payments2003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_puppetrun [18:35:12] RECOVERY - check_puppetrun on frauth2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2001&service=check_puppetrun [18:35:12] RECOVERY - check_puppetrun on civi1002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_puppetrun [18:35:14] RECOVERY - check_puppetrun on frdata1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata1002&service=check_puppetrun [18:35:14] RECOVERY - check_puppetrun on frdev1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_puppetrun [18:35:18] RECOVERY - check_puppetrun on payments1008 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_puppetrun [18:35:18] RECOVERY - check_puppetrun on frmx2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx2001&service=check_puppetrun [18:36:12] RECOVERY - check_puppetrun on pay-lvs2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [18:36:18] RECOVERY - check_puppetrun on frpm1001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpm1001&service=check_puppetrun [18:36:18] RECOVERY - check_puppetrun on payments1006 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_puppetrun [18:36:18] RECOVERY - check_puppetrun on pay-lvs1004 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1004&service=check_puppetrun [18:36:19] RECOVERY - check_puppetrun on frauth1001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth1001&service=check_puppetrun [18:37:12] PROBLEM - check_raid on frbackup2002 is CRITICAL: CRITICAL: PERC 0: physical 12: UGood, physical 13: UGood https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_raid [18:42:18] PROBLEM - check_raid on frbackup2002 is CRITICAL: CRITICAL: PERC 0: physical 12: UGood, physical 13: UGood https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_raid [18:46:50] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10Pcoombe) Okay, I was able to reproduce. This was my payments URL: German language, in Austria, 2 € recurring donation https://payments.wikimedia.org/index.php?title=Special:Adye... [18:47:18] PROBLEM - check_raid on frbackup2002 is CRITICAL: CRITICAL: PERC 0: physical 12: UGood, physical 13: UGood https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_raid [18:50:37] (03CR) 10Damilare Adedoyin: "Thanks jgleeson, I called the method in order to update the request parameter. Please view my response inline." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/811746 (https://phabricator.wikimedia.org/T311625) (owner: 10Damilare Adedoyin) [18:52:18] PROBLEM - check_raid on frbackup2002 is CRITICAL: CRITICAL: PERC 0: physical 12: UGood, physical 13: UGood https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_raid [18:54:07] ACKNOWLEDGEMENT - check_raid on frbackup2002 is CRITICAL: CRITICAL: PERC 0: physical 12: UGood, physical 13: UGood Jeff_Green investigating https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_raid [18:54:12] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10Pcoombe) Okay forget the bit above about µBlock Origin console warnings, I was able to reproduce this again with it disabled. The donation only goes through on the second click... [19:01:23] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) I have sent this info over to Adyen for review. [19:07:18] RECOVERY - check_raid on frbackup2002 is OK: OK: PERC 0: 14 physical, 1 virtual (36.381 TB) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_raid [19:07:52] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10Pcoombe) I didn't see anything similar making a one-time donation. Note that all the CIDs referenced were initially recurring donations too. [19:12:06] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) Noted. Likely why I couldn't reproduce. Mine were all one time with MC after. [19:14:05] 10fundraising-tech-ops: Fundraising servers RAID configuration oddities - https://phabricator.wikimedia.org/T312892 (10Jgreen) [19:14:43] icinga wave fr-tech? [19:15:07] ah backup stuff Jeff_Green ? [19:16:10] jgleeson: replacement of the RAID monitoring software, which surfaced some config weirdness [19:17:00] ah cool [19:18:53] damilare: WRT the deployment. We just need an update to the mediawiki/vendor repo with the new changes pulled in from v0.8.1 of Smashpig and then an update to payments-wiki pulling in your new https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/813285 submodule hash [19:20:47] this is sample commit I added to vendor for an earlier version bump https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/809664 [19:21:34] I saw this earlier jgleeson, just need to confirm a few details with you. [19:22:15] ah cool. watcha need? [19:24:22] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10AMJohnson) If it's helpful below are a few more donor tickets we've received today. Each of the donors accidentally submitted two recurring donations. Haven't seen any screenshot... [19:24:22] there are some files in the vendor dir like the Adyen/Ingenico, is it possible to remove them and reinitialise the vendor dir with what's needed https://i.imgur.com/CDi6Zre.png [19:25:11] ah yeah you can rm -rf that dir [19:25:25] and do `git submodule update --init vendor` [19:25:36] it should restore it back to the original state [19:26:07] oh wait damilare [19:26:16] do you have the bind mount active? [19:26:30] if so, you'll need to disable it for this bit [19:26:50] I had disabled it earlier [19:26:58] ok that's good [19:29:23] it's on a branch that's not rel1_35 [19:29:56] there is attempted documentation here not sure if it would help damilare https://wikitech.wikimedia.org/wiki/Fundraising/Cluster/Deployments [19:30:41] i liked some example patches of previous deploys [19:30:44] linked* [19:31:40] damilare: it'll be checked out on the submodule hash [19:32:35] so you'll need to switch to fundraising/REL1_35 [19:33:10] thanks cstone [19:40:35] 10Fundraising-Backlog, 10FR-Braintree-Integration, 10Documentation: Document Braintree integration - https://phabricator.wikimedia.org/T303452 (10AnnWF) https://wikitech.wikimedia.org/wiki/Fundraising/Data_and_flow/PSP_integrations/Braintree [19:47:13] 10Fundraising-Backlog, 10FR-Braintree-Integration: Add spinner to Braintree UI while waiting for button to appear - https://phabricator.wikimedia.org/T312897 (10AndyRussG) [19:50:45] 10Fundraising-Backlog, 10FR-Braintree-Integration: Add spinner to Braintree UI while waiting for button to appear - https://phabricator.wikimedia.org/T312897 (10AnnWF) [20:08:21] 10Fundraising-Backlog, 10fundraising-tech-ops, 10Privacy Engineering: clear trailing number from email addresses in logs and other places outside civi - https://phabricator.wikimedia.org/T306360 (10Jgreen) [20:10:44] 10Fundraising-Backlog, 10fundraising-tech-ops, 10Privacy Engineering: clear trailing number from email addresses in logs and other places outside civi - https://phabricator.wikimedia.org/T306360 (10Jgreen) process-control logs up to 2022-04-30 are done. 6771 files were clobbered on frlog1001/frlog2001/frback... [20:31:40] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10Pcoombe) p:05Triage→03Unbreak! I did some more digging and wrote a query to find contacts who have made 2 recurring Adyen donations since yesterday from a banner. There are 1... [20:33:45] 10Fundraising-Backlog: new way to handle session dead - https://phabricator.wikimedia.org/T310550 (10SHust) Sharing the case below from a donor that has 4 declined attempts from 07/08 but says they saw the TY page (Ticket# 1136227) @AnnWF Dlocal ref# 2304479015, 2304478657, 2304478508, 2304477520, 2216630235, 2... [20:47:30] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) Escalated to Adyen executives having not heard from Adyen support as of 1:45 Pm Tuesday. [20:58:14] (03PS1) 10Damilare Adedoyin: Update DonationInterface and vendor submodules [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/813333 [21:03:54] 10Fundraising-Backlog, 10fundraising-tech-ops, 10Privacy Engineering: clear trailing number from email addresses in logs and other places outside civi - https://phabricator.wikimedia.org/T306360 (10Jgreen) [21:08:15] (03CR) 10Damilare Adedoyin: [C: 03+2] Update DonationInterface and vendor submodules [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/813333 (owner: 10Damilare Adedoyin) [21:12:46] (03PS2) 10Damilare Adedoyin: Update DonationInterface and vendor submodules [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/813333 [21:24:55] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10AndyRussG) Thanks, all! FR-Tech is digging into this right now, btw... [21:31:09] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) Update from Adyen is as follows, which I have pushed back on as even if WMF is initiating two separate transactions, why would the user be getting an 'expired page' erro... [21:55:16] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10XenoRyet) @EMartin Over here in FR-tech land, we have been able to replicate the bug, and we're currently working on figuring out if it's our side or theirs. [22:00:06] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) Thank you. I am in the process of refunding the duplicates with the payments team. [22:05:46] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) @Fr-Tech. I just forwarded a thread from Adyen with a question. Can you respond there? They found an error int he log Peter sent: For that checkout page I was able to... [22:06:32] 10Fundraising-Backlog: Monthly Convert variant on a recurring donation is pausing on the modal - https://phabricator.wikimedia.org/T312905 (10Cstone) [22:09:01] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10XenoRyet) @EMartin We think we've actually tracked down the problem and the underlying bug. We're writing up the ticket to fix that now, but we should also have a workaround tha... [22:09:56] 10Fundraising-Backlog: Adyen api response with unrecognized parameters - https://phabricator.wikimedia.org/T312906 (10Cstone) [22:21:18] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10XenoRyet) There's an underlying bug regarding having monthly convert both default and hardcoded in a variant, so the workaround is to take out the monthly convert hardcoding out... [22:30:32] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) Thanks @xenoryet. Good to know. I will have Adyen stand down on this. Note to the task: Sandra sent an email to Asjah to have her and I stop refunding as DR wishes t... [22:47:42] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10HNordeenWMF) Ok, I just finished removing hardcoded Adyen and monthlyConvert_011 for credit card from Bundle banners. Here is [[ https://en.wikipedia.org/w/index.php?title=NASA&... [22:54:40] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10TSkaff) I still got an error. Is it possible something's caching? [23:00:38] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) I still am getting errors @HNordeenWMF telling me to contact donor relations. Here is is in German as I used Peter's link and even an alternate and same problem:{F35315559} [23:00:49] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10HNordeenWMF) @TSkaff yes I think it the variants can be sticky! Try incognito? [23:02:20] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) I still see the authorization make it Adyen and the transaction is preparing to settle despite the error too. Same problem.... @XenoRyet @HNordeenWMF [23:03:35] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10XenoRyet) @EMartin The link in your screenshot looks like it still has the monthly convert variant in it. I don't think that should be there for the workaround. [23:03:53] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) I am using incognito when getting these errors fyi [23:04:40] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) I tried from donate wiki as well with the same problem [23:13:34] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10TSkaff) I'm still seeing error msgs as well, on Incognito and using the banner w/o the variant. Are other people testing? [23:23:14] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) Note that Thea is getting errors too, but I see her transactions getting through in Adyen. She THINKs they didn't but they did. Same problem as our users. {F35315580} [23:31:41] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10Cstone) Hi @EMartin , @TSkaff did you try without the variant in an incognito window? I tested and saw it working for me. If you keep seeing issues I can try again in an hour. [23:33:00] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) yes, all of my testing is in incognito mode and without the variant as well. [23:34:07] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10TSkaff) Yes, tried Incognito and using the banner w/o the variant. I'm going to give it a little rest time & try again soon. [23:41:04] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) @Cstone Hi Christine, I was able to get one payment through successfully with a TY page. However I tried again and got this error again: {F35315591} [23:44:35] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10TSkaff) I get some that go through as well. But the frustrating thing is that this was happening to donors 25% of the time, so when we see one go through, we can't conclude the i... [23:48:05] 10Fundraising-Backlog, 10fr-donorservices: Bundle test unintended duplicate donations - https://phabricator.wikimedia.org/T310915 (10EMartin) To reiterate Peter's findings of earlier today, the error occurs when a donor selects give monthly (not give once). Those are the transactions that seem to intermittent...