[07:26:24] <fabfur>	 I'm about to disable varnishkafka (webrequest) on A:cp -> T393772
[07:26:25] <stashbot>	 T393772: Shutdown varnishkafka instances - https://phabricator.wikimedia.org/T393772
[07:26:36] <fabfur>	 eventual alerts are on me!
[07:32:50] <jynus>	 ok, thanks
[07:34:46] <marostegui>	 One of the biggest ISP in Spain (Movistar/o2) is having issues
[07:38:45] <marostegui>	 slyngs: is idp.wikimedia.org having issues?
[07:40:01] <slyngs>	 We just upgrade to version 7.1 are you see any problems?
[07:40:16] <marostegui>	 slyngs: It keeps saying my credentials are not valid :)
[07:40:31] <slyngs>	 Hmm
[07:41:50] <marostegui>	 slyngs: Login against idp.wikimedia.org seems to work now
[07:42:10] <marostegui>	 But when hitting turnilo, and logging via that, it fails
[07:42:42] <slyngs>	 I'll just test
[07:43:18] <marostegui>	 slyngs: So loging via https://idp.wikimedia.org/login?service=https%3a%2f%2fturnilo.wikimedia.org%2f fails, but login via idp directly works
[07:43:53] <slyngs>	 Can you reach Turnilo after signing in? It worked for me, but Turnilo took forever to do anything
[07:44:21] <marostegui>	 slyngs: No, I loging via idp, and it works, and then I go to turnilo and asks me again for credentials via idp
[07:44:33] <marostegui>	 But icinga works, or orchestrator
[07:44:59] <marostegui>	 Maybe something specific with turnilo?
[07:45:12] <slyngs>	 I found you in the log, just a sec.
[07:45:15] <jelto>	 (I don't have any issues so far, idp, turnilo and other internal sites work for me)
[07:45:18] <marostegui>	 o/
[07:46:05] <marostegui>	 Same issue with librenms
[07:46:22] <slyngs>	 I think it's related to you maybe having had an mfa-token at some point
[07:46:38] <marostegui>	 slyngs: What do you mean?
[07:47:28] <slyngs>	 The log complains about you missing a mfa-method
[07:48:14] <marostegui>	 slyngs: How would I fix that? Because I've not done anything apart from loging as I normally do :)
[07:48:27] <slyngs>	 I think that's something I'll fix :-)
[07:48:33] <marostegui>	 :) thanks
[07:50:27] <slyngs>	 Can you test now... I'm not convinced though
[07:50:32] <marostegui>	 checking
[07:50:42] <marostegui>	 same
[08:00:34] <slyngs>	 marostegui: Just a sec
[08:02:21] <slyngs>	 marostegui: Can you try to sign out https://idp.wikimedia.org/logout and then log back in
[10:22:05] <fabfur>	 question for phabricator pro users: how do I create a project/tag? I tried from the https://phabricator.wikimedia.org/project/ page (Create Project) but I have no permission to do so
[10:27:07] <moritzm>	 fabfur: see here, confusingly the docs are not on wikitech, but on mediawiki.org: https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects
[10:27:15] <fabfur>	 ah thanks
[10:36:11] <_joe_>	 fabfur: always search wikitech and mediawiki.org before asking :)
[10:36:27] <_joe_>	 and then ask if the docs are up to date, ofc
[10:36:45] <fabfur>	 searched on wikitech, didnt' thought about the mediawiki.org 
[10:37:52] <p858snake|cloud>	 fabfur: {{done}}
[10:38:37] <fabfur>	 thanks p858snake|cloud !!!
[11:06:07] <topranks>	 folks just a heads up in about 10 mins I'll be depooling codfw in dns, ahead of installing new line cards in our core routers on site (T393552)
[11:06:08] <stashbot>	 T393552: codfw: setup MPC10E-10C and SCBE3 - https://phabricator.wikimedia.org/T393552
[11:09:31] <slyngs>	 Note on the CAS 7.1 upgrade: If you experience being able to authenticate on https://idp.wikimedia.org but being rejected from a service, e.g. Netbox, Alerts, Turnilo or some else, please sign out via https://idp.wikimedia.org/logout This should clear any in compatible state left over from CAS 7.0.
[16:59:07] <inflatador>	 jynus (or anyone else), is this how I disable alerts for my hosts while I'm reimaging? https://gerrit.wikimedia.org/r/c/operations/puppet/+/501368/2/hieradata/hosts/cloudvirt1008.yaml ref https://phabricator.wikimedia.org/T394640#10839636  
[17:02:36] <volans>	 do you expect the host to have alerts after the reimage?
[17:02:45] <mutante>	 inflatador: unless something changed it used to be:  profile::monitoring::notifications_enabled: false
[17:02:55] <sukhe>	 "profile::monitoring::notifications_enabled: false" is what I have, yeah
[17:03:21] <volans>	 IIRC that's only for icinga alerts, or did alertmanager support was added recently?
[17:03:43] <inflatador>	 volans yeah, I'm trying to stop spamming mainline SRE. The unassigned shards alerts are icinga, so that would be good enough. Just wanna make sure I have the right hiera var
[17:05:49] <inflatador>	 Eqiad is depooled and the unassigned shards alerts are expected
[17:06:10] <mutante>	 you can also run the downtime cookbook and set it for a long period and then remove it again later. that should do both icinga and alertmanager afaict
[17:06:23] <volans>	 than when they said, you can define it at different levels if you want to match more hosts
[17:07:43] <inflatador>	 I've been downtiming, but the reimage cookbooks remove the downtime. If it's just a puppet change to the eqiad hiera that's fine. Y'all shouldn't have to see those alerts anyway, I'm working on routing them to DPE SRE instead as well
[17:07:50] <mutante>	 thanks for avoiding the monitoring noise
[17:10:00] <mutante>	 yea, you can use  hieradata/role/eqiad/foo.yaml 
[17:33:24] <inflatador>	 OK. CR up for disabling monitoring for eqiad cirrussearch/elastic if anyone has time to look: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1148402
[17:38:47] <mutante>	 +1, other stuff already uses that contactgroup_name. that was the only part I wanted to check
[17:41:17] <inflatador>	 thanks mutante ! Merging...