[07:02:14] <wikibugs>	 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10Marostegui) m1-master.eqiad.wmnet switched over to dbproxy1012 which is on row A. Once this row is done, we need to revert that.
[07:02:44] <wikibugs>	 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10Marostegui)
[16:41:47] <wikibugs>	 10Traffic: Unexpected auditd service restart failure - https://phabricator.wikimedia.org/T287266 (10ssingh)
[17:07:05] <wikibugs>	 10Traffic, 10SRE: Unexpected auditd service restart failure - https://phabricator.wikimedia.org/T287266 (10ssingh) p:05Triage→03Low
[18:37:20] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: cr2-codfw:fpc0 crash - https://phabricator.wikimedia.org/T287110 (10cmooney) @Papaul replaced card and interfaces have been switched up.  All seems ok. ` cmooney@re0.cr2-codfw> show chassis fpc pic-status 0                       Slot 0   Online       MPCE Type 3...
[18:40:35] <topranks>	 anybody around?
[18:40:56] <topranks>	 Papaul has replaced the faulty card in cr2-codfw and all looks ok, all interfaces back up and trafficing.
[18:41:22] <topranks>	 I am inclined to re-pool eqiad, but unsure the exact best way to proceed.
[18:41:34] <topranks>	 Should I just revert https://gerrit.wikimedia.org/r/c/operations/dns/+/703562 ?
[18:51:34] <XioNoX>	 rzl maybe? as you're clinic duty :)
[18:51:54] <XioNoX>	 I'n on my phone, but it would be nice to repool eqiad before the weekend
[18:52:09] <rzl>	 sure, happy to help
[18:52:14] <topranks>	 Thanks XioNox
[18:52:19] <XioNoX>	 thanks a lot!
[18:52:30] <topranks>	 rzl: great
[18:52:34] <rzl>	 topranks: yep, we'll revert that commit, feel free to send it to me for review
[18:54:44] <rzl>	 topranks: ops/dns works just like the ops/puppet repo, you'll +2 and self-merge
[18:54:55] <rzl>	 then ssh to any authdns server and `sudo authdns-update`
[18:55:36] <topranks>	 cool thanks.
[18:55:54] <topranks>	 above change reverted if you can +1
[18:56:15] <rzl>	 already stamped it, fire when ready
[18:56:26] <topranks>	 ok great thanks :)
[18:57:00] <topranks>	 should I worry it says "merge conflict" on it?
[18:57:25] <rzl>	 it'll probably rebase cleanly if you hit the rebase button in gerrit
[18:57:38] <rzl>	 not a whole lot happening in that file lately, except for adding and removing this line repeatedly
[18:58:04] <topranks>	 ok let's see.  thanks.
[18:59:07] <rzl>	 yep, perfect
[18:59:16] <rzl>	 wait for jenkins to V+2 it again, and then you can go ahead
[19:00:05] <topranks>	 all looks good with that thanks
[19:14:57] <jinxer-wm>	 (VarnishTrafficDrop) firing: 61% GET drop in text@ during the past 30 minutes  - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org
[19:34:34] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: cr2-codfw:fpc0 crash - https://phabricator.wikimedia.org/T287110 (10cmooney) Everything still looking good, eqiad re-pooled and combined stats across sites as they were but eqiad back in the pool.  Resolving task.
[19:34:56] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: cr2-codfw:fpc0 crash - https://phabricator.wikimedia.org/T287110 (10cmooney) 05Open→03Resolved
[19:49:57] <jinxer-wm>	 (VarnishTrafficDrop) resolved: 63% GET drop in text@ during the past 30 minutes  - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org