[09:08:53] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Traffic Engineering for Anycast Ranges - https://phabricator.wikimedia.org/T288843 (10ayounsi) I currently assume that: * IX peers are mostly local, so no special care needs to happen to them ** If this happens to be incorrect we could inv... [09:13:52] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Traffic Engineering for Anycast Ranges - https://phabricator.wikimedia.org/T288843 (10jbond) lgtm > using the NO-EXPORT BGP community (most likely not supported by many peers) FYI i have had a good experience using no-export at IX's, i.e.... [09:31:15] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Traffic Engineering for Anycast Ranges - https://phabricator.wikimedia.org/T288843 (10cmooney) >> IX peers are mostly local, so no special care needs to happen to them >> >> - If this happens to be incorrect we could investigate not sendin... [14:23:40] 10Puppet, 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Jelto) I imported the latest data to `gitlab2001` and everything except ssh looks fine. I prepared a patch ([728380](https://... [15:43:24] 10Puppet, 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Jelto) Puppet on `gitlab1001` is enabled again and the puppet run was successful. Web interface works, pulling over ssh works... [17:05:39] 10Puppet, 10Infrastructure-Foundations, 10SRE: Puppet: tlsproxy localssl default_server make a Notify at each run - https://phabricator.wikimedia.org/T191393 (10BBlack) #Traffic doesn't use `tlsproxy::localssl` anymore (since quite some time ago!), so we don't really have an opinion on its remaining use or m... [17:31:00] 10Puppet, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE, 10Technical-Debt: Uniform cluster nomenclature across puppet - https://phabricator.wikimedia.org/T159411 (10BBlack) [17:59:51] 10netops, 10Infrastructure-Foundations, 10Pybal, 10SRE, 10Traffic: Rename lvs* LLDP port descriptions after upgrading to stretch - https://phabricator.wikimedia.org/T192087 (10BBlack) [18:00:01] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic: cr1-eqsin 4 onboard interfaces down - https://phabricator.wikimedia.org/T193897 (10BBlack) [18:00:09] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic: Aug 28th: turn off 1/3 esams-knams lasers in advance of Relined PA-988002 maintenance - https://phabricator.wikimedia.org/T230448 (10BBlack) [18:00:18] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10ops-codfw: Interface errors on asw-d-codfw:xe-2/0/47 - https://phabricator.wikimedia.org/T193677 (10BBlack) [18:00:30] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10observability: Network port utilization alerts should be paging - https://phabricator.wikimedia.org/T224888 (10BBlack) [18:00:38] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic: BGP: Investigate isolating codfw and eqiad - https://phabricator.wikimedia.org/T246721 (10BBlack) [18:00:46] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic: Wikimedia projects not reachable for some Telecom Italia users - https://phabricator.wikimedia.org/T262869 (10BBlack) [18:00:50] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10Patch-For-Review: Remove multicast - https://phabricator.wikimedia.org/T257573 (10BBlack) [18:01:24] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10Goal: Increase network capacity (2018-19 Q2 Goal) - https://phabricator.wikimedia.org/T207668 (10BBlack) [18:01:32] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10Patch-For-Review: ulsfo <-> codfw transit link flapping causing nginx availability alerts - https://phabricator.wikimedia.org/T219591 (10BBlack) [18:01:41] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10Wikimedia-Incident: Configure interface damping on primary links - https://phabricator.wikimedia.org/T196432 (10BBlack) [18:01:47] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10Patch-For-Review: Free up 185.15.59.0/24 - https://phabricator.wikimedia.org/T211254 (10BBlack) [18:01:55] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic: IPv6 ~20ms higher ping than IPv4 to gerrit - https://phabricator.wikimedia.org/T211079 (10BBlack) [18:02:10] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10Patch-For-Review: cp intermittent IPsec MTU issue - https://phabricator.wikimedia.org/T195365 (10BBlack) [18:02:20] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10ops-ulsfo: troubleshoot cr3/cr4 link - https://phabricator.wikimedia.org/T196030 (10BBlack) [18:02:30] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10Patch-For-Review: Offload pings to dedicated server - https://phabricator.wikimedia.org/T190090 (10BBlack) [18:02:36] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10Patch-For-Review: eqiad row D switch upgrade - https://phabricator.wikimedia.org/T172459 (10BBlack) [18:27:19] 10netbox, 10DNS, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): Move some of wikimediacloud.org 185.15.56.0/23 to Netbox - https://phabricator.wikimedia.org/T268621 (10BBlack) [18:33:41] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic: Register as14907 dot net (or other similar domain) for network infra concerns - https://phabricator.wikimedia.org/T292866 (10CDanis) [18:36:03] 10netbox, 10netops, 10DNS, 10Infrastructure-Foundations, and 2 others: Cloud: define relationship between wikimediacloud.org domain, CIDR prefixes and netbox automation - https://phabricator.wikimedia.org/T266331 (10BBlack) [18:41:16] 10netops, 10Data-Services, 10Infrastructure-Foundations, 10SRE, and 2 others: wikireplicas last-minute infra work to discuss / resolve - https://phabricator.wikimedia.org/T273248 (10BBlack) 05Open→03Resolved a:03ayounsi [18:54:10] 10netops, 10Infrastructure-Foundations, 10SRE: TATA SKY Broadband (AS134674) issues with connecting to upload.wikimedia.org - https://phabricator.wikimedia.org/T275234 (10BBlack) Removing #Traffic as I don't think this looks actionable for our team (but might still be for netops if the conversations above ar... [19:00:49] 10Puppet, 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Dzahn) No concerns and nice work, Jelto. 👍 [19:01:00] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10procurement: drmrs: primary software task - https://phabricator.wikimedia.org/T282788 (10BBlack) [19:01:51] 10Puppet, 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10brennen) Login to gitlab.wikimedia.org seems to be broken for 2fa users currently (recurring prompt for 2fa code after authen... [19:05:47] 10Puppet, 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10AntiCompositeNumber) [19:11:42] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic: externally-hosted NEL report forwarders for more timely report reception - https://phabricator.wikimedia.org/T292870 (10CDanis) p:05Triage→03Low [19:12:00] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic: externally-hosted NEL report forwarders for more timely report reception - https://phabricator.wikimedia.org/T292870 (10CDanis) [19:16:22] 10Puppet, 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10brennen) I think this is the culprit: ` brennen@gitlab1001:~$ sudo grep session_duration /opt/gitlab/embedded/service/gitlab... [19:24:41] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic: externally-hosted NEL report forwarders for more timely report reception - https://phabricator.wikimedia.org/T292870 (10CDanis)