[12:27:09] <_joe_>	 !incidents
[12:27:10] <sirenbot>	 2950 (ACKED)  [FIRING:1] ProbeDown (10.2.2.53 ip4 thanos-query:443 probes/service http_thanos-query_ip4 ops page eqiad prometheus sre)
[12:37:26] <paravoid>	 hey folks - check your inbox for an email to tech-all/product-all with some important news
[12:53:36] <apergos>	 nice job of understatement there ;-)
[12:59:17] <paravoid>	 :P
[13:36:46] <marostegui>	 In 25 minutes I am switching m5 db master
[13:45:16] <Krinkle>	 cdanis: low priority, but I'd love to pair on a sprint at some point to iteratively sync the dbctl value for MW such that we phase out most or all the post processing people added in 2y in wmf-config.
[13:46:07] <Krinkle>	 Makes things a bit more transparent and easier to debug. Right now it's a fairly odd shape that isn't simply a subset or superset of what Mw needs
[14:23:47] <cdanis>	 Krinkle: yeah, it was built around automating DBA workflows, with basically everything else as a secondary concern
[15:37:08] <_joe_>	 I am doing a series of potentially dangerous changes to the apache configuration for mediawiki
[15:37:19] <_joe_>	 so I have disabled puppet everywhere 
[15:42:26] <_joe_>	 (on mw hosts, that is)
[16:24:30] <vgutierrez>	 🍿
[16:29:50] <_joe_>	 vgutierrez: collateral damage could include noc.w.o and wikitech
[16:41:07] <_joe_>	 wikitech seems safe
[16:45:28] <_joe_>	 noc.w.o seems too
[16:49:36] <_joe_>	 now propagating across mw nodes
[16:53:31] <sukhe>	 _joe_: $deityspeed!
[16:53:49] <_joe_>	 the flying spaghetti monster is protecting me, as always
[16:54:26] <sukhe>	 hopefully not Barilla spaghetti :P
[17:53:05] <_joe_>	 I'll take a break to let the train run
[17:53:09] <_joe_>	 but things look ok
[18:30:29] <_joe_>	 !incidents
[18:30:29] <sirenbot>	 2951 (ACKED)  [FIRING:1] ProbeDown (10.2.2.40 ip4 labweb-ssl:7443 probes/service http_labweb-ssl_ip4 ops page eqiad prometheus sre)
[18:30:30] <sirenbot>	 2950 (RESOLVED)  [FIRING:1] ProbeDown (10.2.2.53 ip4 thanos-query:443 probes/service http_thanos-query_ip4 ops page eqiad prometheus sre)
[18:31:29] <_joe_>	 !resolve 2951
[18:31:30] <sirenbot>	 2951 (ACKED)  [FIRING:1] ProbeDown (10.2.2.40 ip4 labweb-ssl:7443 probes/service http_labweb-ssl_ip4 ops page eqiad prometheus sre)
[18:31:37] <_joe_>	 uhm
[18:31:39] <_joe_>	 no heh
[18:31:44] <_joe_>	 !incidents
[18:31:45] <sirenbot>	 2951 (ACKED)  [FIRING:1] ProbeDown (10.2.2.40 ip4 labweb-ssl:7443 probes/service http_labweb-ssl_ip4 ops page eqiad prometheus sre)
[18:31:45] <sirenbot>	 2950 (RESOLVED)  [FIRING:1] ProbeDown (10.2.2.53 ip4 thanos-query:443 probes/service http_thanos-query_ip4 ops page eqiad prometheus sre)