[04:18:05] 10netops, 10DC-Ops, 10SRE: allow mgmt network to access tftp servers for firmware updates - https://phabricator.wikimedia.org/T283771 (10Papaul) @RobH I have only one question for now. what is or will be your approach on keeping the TFTP server up to date with the latest firmware. [07:25:01] 10netops, 10DC-Ops, 10SRE: allow mgmt network to access tftp servers for firmware updates - https://phabricator.wikimedia.org/T283771 (10ayounsi) As you said it would be a good idea to see how it fits in the big automation picture. First by detailing precisely the current workflows, identifying the pain poin... [10:35:51] 10CFSSL-PKI, 10SRE, 10Patch-For-Review: Investigate Check for expired certificates debmonitor - https://phabricator.wikimedia.org/T283185 (10jbond) 05Open→03Resolved Closing, it was decided to remove this check as there are too many variables to make it useful, further we already have expiry checking for... [12:49:10] 10netops, 10SRE: Netbox has incorrect email address for GTT - https://phabricator.wikimedia.org/T246564 (10ayounsi) 05Open→03Resolved Thanks all set and updated. noc@gtt.net is a valid email according do their website (so maybe it was a temporary issue?), and I added their 2nd level escalation email as we... [12:59:17] 10netops, 10SRE: Netbox has incorrect email address for GTT - https://phabricator.wikimedia.org/T246564 (10ayounsi) Actually looks like they don't want emails. So I left a note in Netbox saying that it's phone or portal only. [13:10:16] 10netbox, 10Patch-For-Review: Netbox: import from PuppetDB fails on a specific host - https://phabricator.wikimedia.org/T276760 (10Volans) 05Open→03Resolved The fix has been deployed, the script runs correctly now. Resolving [13:12:36] 10netbox: netbox import from PuppetDB: it should ignore 192.168.0.0/16 - https://phabricator.wikimedia.org/T283813 (10Volans) p:05Triage→03Medium [15:36:36] 10netops, 10DC-Ops, 10SRE: allow mgmt network to access tftp servers for firmware updates - https://phabricator.wikimedia.org/T283771 (10RobH) >>! In T283771#7118281, @Papaul wrote: > @RobH > I have only one question for now. what is or will be your approach on keeping the TFTP server up to date with the la... [15:46:18] 10homer: Homer CI: verify Junos syntax - https://phabricator.wikimedia.org/T253194 (10ayounsi) Opened an upstream task with some questions: https://github.com/codeout/junoser/issues/24 Indeed, not having CI able to pull data from Netbox is going to be an issue down the road. Current workaround is to maybe use n... [16:00:23] 10netbox, 10DC-Ops: Various netbox alerts running for days - https://phabricator.wikimedia.org/T283483 (10Cmjohnson) [16:20:19] 10Puppet, 10User-jbond: Upgrade puppet to use hiera version 5 - https://phabricator.wikimedia.org/T254248 (10jbond) 05Open→03Resolved [16:24:21] 10CAS-SSO, 10SRE, 10User-jbond: Cross data center setup for CAS - https://phabricator.wikimedia.org/T233931 (10jbond) [16:25:06] 10CAS-SSO, 10SRE, 10Patch-For-Review, 10User-jbond: CAS Store U2f tokens in a database - https://phabricator.wikimedia.org/T256113 (10jbond) 05Open→03Resolved a:03jbond [16:25:15] 10Puppet, 10SRE, 10Patch-For-Review, 10User-jbond: Add check for changes applied at all runs - https://phabricator.wikimedia.org/T242910 (10jbond) [16:25:57] 10Puppet, 10SRE, 10Patch-For-Review, 10User-jbond: reprepo user different on release1001 and release2001 - https://phabricator.wikimedia.org/T245612 (10jbond) 05Open→03Resolved a:03jbond [16:26:15] 10Puppet, 10SRE, 10User-jbond: configure and Test vaults capabilities as an ondemand CA - https://phabricator.wikimedia.org/T247509 (10jbond) 05Open→03Resolved [16:26:42] 10CAS-SSO, 10SRE, 10User-jbond: Cross data center setup for CAS - https://phabricator.wikimedia.org/T233931 (10jbond) [16:26:45] 10CAS-SSO, 10SRE, 10Security-Team, 10User-jbond: Further steps for CAS/web SSO - https://phabricator.wikimedia.org/T233921 (10jbond) [16:27:07] 10CAS-SSO, 10SRE, 10Patch-For-Review, 10User-jbond: Replicated ticket registry - https://phabricator.wikimedia.org/T233933 (10jbond) 05Open→03Resolved a:03jbond [16:28:22] 10CAS-SSO, 10SRE, 10Security-Team, 10User-jbond: Further steps for CAS/web SSO - https://phabricator.wikimedia.org/T233921 (10jbond) [16:28:59] 10CAS-SSO, 10SRE, 10Patch-For-Review, 10User-jbond: Add U2F/FIDO as second factor for CAS - https://phabricator.wikimedia.org/T233937 (10jbond) 05Open→03Resolved a:03jbond closing this i think the idea to support multiple options is out of scope of this task [16:30:05] 10CAS-SSO, 10SRE, 10User-jbond: IDP failover improvments - https://phabricator.wikimedia.org/T268217 (10jbond) IdP no longer has the primary/secondery hiera addresses however we should move the services to use a DNS discovery address [16:35:47] 10puppet-compiler, 10SRE, 10User-jbond: populate puppetdb fails for unknown hosts - https://phabricator.wikimedia.org/T248689 (10jbond) 05Open→03Resolved a:03jbond This is now fixed by running first with dev/null [16:37:53] 10Puppet, 10SRE, 10User-jbond: puppetmaster: clean up instances of the puppet-master package - https://phabricator.wikimedia.org/T276339 (10jbond) 05Open→03Resolved a:03jbond [16:39:04] topranks: i just cam accross this task https://phabricator.wikimedia.org/T270391 (i think you mentioned looking at something simlar for azure) [16:41:10] 10Puppet, 10SRE, 10observability, 10User-jbond: PuppetDB grafana graphs not matching logs - https://phabricator.wikimedia.org/T265649 (10jbond) 05Open→03Resolved a:03jbond I made some changes to the metrics graphed and theses are looking much more accurate now [16:42:17] 10CAS-SSO, 10SRE, 10User-jbond: Apereo CAS expose CASCookieSameSite via profile::idp::client::http - https://phabricator.wikimedia.org/T264605 (10jbond) did this make it out? [16:44:49] I [16:45:37] jbond: thanks for the link that's great [16:45:49] np [16:45:54] I banged out a quick script for it but I wasn't sure if it was going to be any use. I'll dig into the task now. [16:48:06] 10Puppet, 10SRE, 10Patch-For-Review, 10User-jbond: hiera_lookup failing to preform lookups after hiera5 upgrade - https://phabricator.wikimedia.org/T258931 (10jbond) 05Open→03Resolved should now use `sudo puppet lookup` [16:50:44] 10Puppet, 10SRE, 10Patch-For-Review, 10User-jbond: puppetise pupet server copy of the public ca.pem - https://phabricator.wikimedia.org/T256721 (10jbond) 05Open→03Resolved [16:50:46] 10Puppet, 10SRE, 10Patch-For-Review, 10User-jbond: Extend Puppet CA Expiry date - https://phabricator.wikimedia.org/T236277 (10jbond) [16:53:18] 10CFSSL-PKI, 10User-jbond: Package Latest version go-cfssl - https://phabricator.wikimedia.org/T283840 (10jbond) p:05Triage→03Medium [16:56:44] ok im out, enjoy the long weekend all, heres hoping for a quite one :) [16:58:40] cheers [17:06:11] 10netops, 10SRE, 10Traffic, 10User-jbond: varnish filtering: should we automatically update public_cloud_nets - https://phabricator.wikimedia.org/T270391 (10cmooney) There is this script for AWS that @ema pointed me towards: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/pro... [17:46:43] 10SRE-tools: debmonitor: 500 on upgraded+downgraded package - https://phabricator.wikimedia.org/T282529 (10Volans) The issue has been fixed and the latest debmonitor-client version released. I'm waiting for the full deploy across the fleet before closing this task though. [18:43:23] 10netops, 10DC-Ops, 10SRE: allow mgmt network to access tftp servers for firmware updates - https://phabricator.wikimedia.org/T283771 (10RobH) I think either A or C, B seems problematic and allows for one person to serve as a blocker for updates being timely. Also I fear that B would make me the single poin... [18:45:08] 10netops, 10DC-Ops, 10SRE: allow mgmt network to access tftp servers for firmware updates - https://phabricator.wikimedia.org/T283771 (10RobH) >>! In T283771#7118462, @ayounsi wrote: > As you said it would be a good idea to see how it fits in the big automation picture. > First by detailing precisely the cur... [20:31:30] 10Packaging, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Create a debian package for Apache Airflow - https://phabricator.wikimedia.org/T277012 (10Ottomata) @Volans42 I've manually reinstalled our dev .deb on an-test-coord1001. What makes it needing to be reinstalled? I can remove it again if ne... [22:01:52] 10Packaging, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Create a debian package for Apache Airflow - https://phabricator.wikimedia.org/T277012 (10Volans) @Ottomata nothing is needed AFAICT, APT is happy again, thanks.