[09:51:26] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10cmooney) Thanks for the detail @aborrero Looking at the setup the logical thing is to allocate the public IPs for these hosts from t... [11:14:43] 10Traffic, 10MW-on-K8s, 10Performance-Team, 10SRE, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10jijiki) >>! In T290536#7383383, @akosiaris wrote: >>>! In T290536#7383272, @jijiki wrote: > > > That's currently my preferred way cause it's determinis... [16:29:57] (VarnishTrafficDrop) firing: 67% GET drop in text@eqsin during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [16:34:57] (VarnishTrafficDrop) resolved: 67% GET drop in text@eqsin during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [16:46:39] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10cmooney) Thanks for the time in the meeting today to discuss. From our chat and a few other things I've looked at we can say: - Thes... [17:01:59] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10aborrero) Pretty much agree with everything you commented @cmooney Just a couple of clarifications: * the servers primary hostname wo... [17:03:03] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10RobH) [17:03:47] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10RobH) [17:04:09] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10RobH) [17:28:20] 10netops, 10Infrastructure-Foundations: Netbox info missing on some WMCS elements - https://phabricator.wikimedia.org/T292097 (10cmooney) [17:28:36] 10netops, 10Infrastructure-Foundations: Netbox info missing on some WMCS elements - https://phabricator.wikimedia.org/T292097 (10cmooney) p:05Triage→03Lowest a:03cmooney [17:50:20] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE: (Need By: TBD) rack/setup/install atlas-codfw.wikimedia.org - https://phabricator.wikimedia.org/T273114 (10cmooney) Seems like we have success :) Port is now up and MAC address learnt: ` cmooney@asw-a-codfw> show ethernet-switching table | match 1/... [18:01:15] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10cmooney) Ok great @aborrero thanks for clarifying. That all 100% fits what I had in mind, so we are on the same page. I'll discuss w... [18:28:28] \o i'm adding an entry to the puppet private repo and I see uncommitted changes for `webserver-misc-apps.discovery` [18:29:06] are these supposed to be committed? I'm changing a different file obv so I'll just leave them unstaged and just commit only my own changes, but figured I'd see if the dirty working directory was expected [18:29:17] https://www.irccloud.com/pastebin/uFhNouqR/ [19:08:25] ryankemper: a dirty working directory is never expected there [19:08:54] cdanis: yeah that was my assumption, especially since the docs say to `sudo git commit -a` (which I avoided for the change I'm adding) [19:09:05] ryankemper: if you have a moment to do some digging around in either corresponding changes to the public repo or to the sudoers log on the puppetmaster it'd be appreciated [19:09:25] I even thought there was once an icinga alert about a dirty checkout there but maybe I'm misremembering [19:10:42] cdanis: sure, will do. thus far all I've done was git blame just to find who had touched the file last in the commit history to get an idea of ownership (confirming my hunch that it was traffic), but haven't x-referenced with the public repo [19:12:12] yeah that looks like a patch made by https://wikitech.wikimedia.org/wiki/Cergen [19:17:38] cdanis: yeah, which is weird because I'm not seeing any `/usr/bin/cergen` invocations in the `/var/log/auth.log`...and IIRC when I've done cergen stuff I had to run the command w/ sudo for it to work [19:29:42] cdanis: okay this must be related to commit `cd79b989a5eae9edf3c1b68dc02a4a1aa2c63007` subject:`(jmm) webserver_misc_apps: Add os-reports.w.o to the SNI list` [19:29:55] I think auth.log doesn't show the cergen since it was ran inside a `sudo bash` perhaps [19:30:29] anyway I assume I should just commit the changes? looks like these files should have been included in `cd79b989a5eae9edf3c1b68dc02a4a1aa2c63007` [19:38:18] ryankemper: yeah that sounds good to me [19:38:20] moritzm: was that you? [20:06:44] okay committed the changes (e9c7a2a8bd5c3554dcf6e14536ea4c68ccfe6240) [20:58:19] ryankemper: oh, sorry, that was in fact me. I thought I had commited them, but sounds like I missed it. Thanks for fixing it up [20:58:43] moritzm: no worries!