[00:31:04] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cumin1002:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [03:11:05] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [04:11:05] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [04:31:04] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cumin1002:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [05:35:15] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Enable gNMI on SRX devices and fasw - https://phabricator.wikimedia.org/T390052#11127323 (10ayounsi) After a long wait Juniper says its not a bug, but they never implemented the feature : > Following discussions with our Engineering team, I’d like t... [05:44:46] 10netops, 06Infrastructure-Foundations, 06SRE: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#11127339 (10ayounsi) Good job!! Next step is to remove OSPF, feel free to send a patch in that regard. It can restore the previously removed `replace: ospf {}` as well as re... [08:31:04] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cumin1002:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [11:59:11] 10CAS-SSO, 10Bitu, 06Infrastructure-Foundations: Logging out of idm does not log me out - https://phabricator.wikimedia.org/T392350#11128129 (10Arendpieter) >>! In T392350#10759377, @SLyngshede-WMF wrote: > The bug should be fixed in CAS, but requires an update. @SLyngshede-WMF Just to clarify: are you sayi... [12:26:15] 10CAS-SSO, 10Bitu, 06Infrastructure-Foundations: Logging out of idm does not log me out - https://phabricator.wikimedia.org/T392350#11128226 (10SLyngshede-WMF) @Arendpieter Yes, that's the same issue, and you are correct that the fix needs to go into the cas-overlay-template repo as it's an update/bugfix in... [12:31:04] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cumin1002:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [12:32:46] the above seems related to the homer/public git::clone operation [12:33:37] executed successfully (corrective) [12:33:54] modules/git/manifests/clone.pp:227 [12:40:20] XioNoX, topranks: we have again local modifications on /srv/homer/public [12:43:22] and are the file modes, maybe it's a conflict with git pull setting them with the execution bit and puppet reverting the bit due to mode => '0440' being se? [12:43:25] *set [13:17:11] we shoudn't, I certainly never modify anything in there..... maybe with Arzehl's testing? [13:49:47] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Remove static routes for LVS VIPs from core routers - https://phabricator.wikimedia.org/T300877#11128565 (10ssingh) Hi folks: Picking this ticket up as part of Traffic cleaning up our stuff. It seems the current static routes -- even if they matter... [13:58:55] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Remove static routes for LVS VIPs from core routers - https://phabricator.wikimedia.org/T300877#11128591 (10cmooney) >>! In T300877#11128565, @ssingh wrote: > It seems the current static routes -- even if they matter at this stage -- are incorrect s... [14:07:04] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Remove static routes for LVS VIPs from core routers - https://phabricator.wikimedia.org/T300877#11128621 (10ayounsi) +1 to remove them :) [14:26:08] 10CAS-SSO, 06Infrastructure-Foundations: Logging out of idm does not log me out - https://phabricator.wikimedia.org/T392350#11128760 (10Arendpieter) [15:04:27] nop, not me [15:04:35] I would never dare :) [15:05:32] as I said earlier I think is a conflict between the puppet definition (mode => 0440) and the newly added scripts that I guess have the execution bit set in git [15:06:50] yeah [15:07:15] so not manually generated [15:54:14] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Remove static routes for LVS VIPs from core routers - https://phabricator.wikimedia.org/T300877#11129257 (10ssingh) Hi Netops folks: Thanks for your feedback. Following up again after discussing this with Traffic. We decided that we will do away wi... [16:31:04] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cumin1002:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [19:08:37] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 06Traffic: Spicerack's Icinga module should provide a way to skip specific services in sub-optimal but desired state - https://phabricator.wikimedia.org/T392848#11129910 (10ssingh) 05Open→03Resolved a:03ssingh Thanks a lot to @elukey and @Vo... [20:31:04] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cumin1002:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange