[08:32:46] 10netops, 10SRE, 10Traffic: Unable to load en.wikipedia.org from 84.19.61.192/26 - https://phabricator.wikimedia.org/T279503 (10A189605) Can you possibly explain why our ISP's interface connected to our network (using IP 84.19.61.194) can successfully ping 91.198.174.192 and 91.198.174.208 (used earlier in t... [09:31:55] 10CAS-SSO, 10SRE, 10User-jbond: Document IDP MFA policy and processes - https://phabricator.wikimedia.org/T284725 (10jbond) p:05Triage→03Medium [09:33:36] anyone made a change to the cumin server puppet policy that may cause this error https://phabricator.wikimedia.org/P16403 [09:33:43] dont see anything obvious but im investigating [09:34:23] * volans checking [09:35:28] volans: never mind i think i have the issue [09:35:42] it was added by jayme [09:35:48] teh file [09:36:01] yeah, me, me [09:41:40] 10CAS-SSO, 10SRE, 10User-jbond: Document IDP MFA policy and processes - https://phabricator.wikimedia.org/T284725 (10Volans) If I may add to the wish list, support multiple tokens for those that have more than one for added redundancy. [09:41:43] fixed [09:42:13] confirmed [09:44:04] sorry! [09:58:22] 10CAS-SSO, 10SRE, 10User-jbond: Document IDP MFA policy and processes - https://phabricator.wikimedia.org/T284725 (10MoritzMuehlenhoff) >>! In T284725#7148119, @Volans wrote: > If I may add to the wish list, support multiple tokens for those that have more than one for added redundancy. For U2F that's curre... [10:15:03] 10Puppet, 10SRE-tools, 10SRE, 10User-jbond: Private puppet commit hook checks current state of folder, not what is staged - https://phabricator.wikimedia.org/T278187 (10JMeybohm) > I had a file that initially failed yamllint, but when I fixed it, I forgot to stage the change, so I didn't actually commit t... [10:21:30] 10Puppet, 10SRE-tools, 10SRE, 10User-jbond: Private puppet commit hook checks current state of folder, not what is staged - https://phabricator.wikimedia.org/T278187 (10Volans) @JMeybohm Just to clarify the chain of events: - you did change the file locally with the typo - staged for commit - commit faile... [10:24:27] 10Puppet, 10SRE-tools, 10SRE, 10User-jbond: Private puppet commit hook checks current state of folder, not what is staged - https://phabricator.wikimedia.org/T278187 (10JMeybohm) >>! In T278187#7148190, @Volans wrote: > Is that correct? Absolutely. [10:27:02] heads up all just going to switch netbox-next back to ldap just to test we can easily switch between the two [10:27:40] 10Puppet, 10SRE-tools, 10SRE, 10User-jbond: Private puppet commit hook checks current state of folder, not what is staged - https://phabricator.wikimedia.org/T278187 (10Volans) Ok, that explains what happened, thanks! As for the solution I think we should just pass the staged files to yamllint instead of t... [10:27:45] jbond: ack, thx! [11:03:39] 10netops, 10SRE, 10Traffic: BGP Policy on aggregate routes prevents them being created in some circumstances. - https://phabricator.wikimedia.org/T283163 (10cmooney) Ok configuration has been added to cr1-eqiad, cr2-eqiad and cr2-codfw (routers with transport links to eqord). Looks to have been successful.... [12:23:58] fyi all i have mertged the contacts metdata change so we can test and iterate with it. all machines will now have a (mostly useless) /etc/wikimedia/contacts.yaml file. And the sretest roles has one with data populated with something a bit more real world [12:28:34] nice! [12:30:43] no ignore that having to revert as there seems to be a tuby issue with the concat module [12:34:01] jbond: ping me when you figured it out and I'll populate it for some of my stuff [12:34:30] XioNoX: yes will do thanks [12:45:24] 10netops, 10SRE, 10Traffic: Unable to load en.wikipedia.org from 84.19.61.192/26 - https://phabricator.wikimedia.org/T279503 (10cmooney) Thus far we have: 1. Validated your IP range or any subset thereof is not on any ban or block lists. 2. Validated we can route from our front-end load-balancer IPs to your... [13:11:27] the contacts stuff is deployed again (XioNoX) [13:13:03] jbond: https://gerrit.wikimedia.org/r/c/operations/puppet/+/699179 is what to duplicate to other roles/profiles? [13:13:26] XioNoX: yes [13:13:31] cool! [13:19:46] jbond: for the names you used phab handles? [13:20:55] XioNoX: i did but there is nothing to enforce that at the moment [13:23:46] good enough for now, thx [13:34:02] jbond: https://gerrit.wikimedia.org/r/c/operations/puppet/+/699209 as a first test [13:36:40] 10netops, 10SRE, 10Traffic, 10Patch-For-Review: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10ssingh) > INFO:homer.transports.junos:Committing the configuration on cr4-ulsfo.wikimedia.org > INFO:homer:Homer run completed successfully on 2 devices:... [13:36:53] XioNoX: +1 lgtm [13:43:26] 10netops, 10SRE, 10Traffic, 10Patch-For-Review: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10cmooney) I can confirm the 185.71.138.0/24 prefix is now being announced to peers from ulsfo, for example: ` cmooney@cr4-ulsfo> show route advertising-pro... [13:47:49] XioNox: I should probably add myself to a few of those I'm thinking. [13:48:11] topranks: for sure [13:48:20] should we put netops instead? [13:48:26] or rename that to SRE I/F network :D [13:48:45] roles are team, profiles are individuals [13:49:04] I'm wondering what's the plan if someone leaves :D [13:49:25] who would dare? [13:51:28] volans: I could imagine something like in the outboarding process there is a step to see if that person is listed as contact. Then remove them and open a task to replace them with a new person [13:52:07] (out for a run, back in 40min) [13:53:14] k [14:32:46] 10netops, 10SRE, 10Traffic, 10Patch-For-Review: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10ssingh) > INFO:homer.transports.junos:Committing the configuration on cr2-eqiad.wikimedia.org > INFO:homer:Homer run completed successfully on 2 devices:... [14:37:18] 10netops, 10SRE, 10Traffic, 10Patch-For-Review: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10ssingh) Additional confirmation, since I am enjoying the reduced latency of the new Toronto -> eqiad route instead of the old Toronto -> codfw :) ` kdig... [14:37:59] 10netops, 10SRE, 10Traffic, 10Patch-For-Review: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10cmooney) Yep! Seeing very nice latency from NY to wikidough now :) ` root@nyc2:~# mtr -b -w -z -c 5 185.71.138.138 Start: 2021-06-10T16:35:02+0200 HOST:... [15:21:22] 10netops, 10DC-Ops, 10SRE: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10RobH) [15:25:02] 10SRE-tools, 10netops, 10DC-Ops, 10SRE: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10Volans) I'm not sure yet how the automation side of things will look like, but there is a good chance that it could use redfish. In... [15:56:31] 10netops, 10SRE, 10Traffic: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10ssingh) 05Open→03Resolved a:03ssingh Marking this as resolved as we have completed all the intended tasks for now and the routers have been configured. On our (Traffic's... [15:59:58] 10SRE-tools, 10netops, 10DC-Ops, 10SRE: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10ayounsi) From IRC conversation: We're going to do a 1 off to ease DCops pain of upgrading a large amount of firmwares. Once those 40... [17:07:31] 10SRE-tools, 10netops, 10DC-Ops, 10SRE: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10RobH) a:03jbond @jbond & @MoritzMuehlenhoff: Would it be ok for me to temp push the Dell firmware files to our install server via... [17:20:55] 10SRE-tools, 10netops, 10DC-Ops, 10SRE: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10MoritzMuehlenhoff) >>! In T283771#7149531, @RobH wrote: > @jbond & @MoritzMuehlenhoff: > > Would it be ok for me to temp push the D... [17:53:45] 10SRE-tools, 10netops, 10DC-Ops, 10SRE: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10RobH) >>! In T283771#7149561, @MoritzMuehlenhoff wrote: >>>! In T283771#7149531, @RobH wrote: >> @jbond & @MoritzMuehlenhoff: >> >>... [17:59:02] 10SRE-tools, 10netops, 10DC-Ops, 10SRE: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10MoritzMuehlenhoff) >>! In T283771#7149663, @RobH wrote: >> Which size are these files? That's fine, if it's not more than say 5 G, t...