[07:35:19] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Modify homer/automation templates to support 100BaseTX interfaces with autoneg disabled. - https://phabricator.wikimedia.org/T288343 (10cmooney) p:05Triage→03Low [07:35:27] 10netops, 10Infrastructure-Foundations, 10SRE: Traffic Engineering for Anycast Ranges - https://phabricator.wikimedia.org/T288843 (10cmooney) p:05Triage→03Medium [07:38:31] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): Join ARIN waiting list to request additional IPv4 resources. - https://phabricator.wikimedia.org/T288342 (10cmooney) p:05Triage→03Low [07:38:43] 10netops, 10Infrastructure-Foundations, 10SRE: Create an alert for output discards on network devices - https://phabricator.wikimedia.org/T284593 (10cmooney) p:05Triage→03Medium [10:44:04] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, and 2 others: LLDP: Ganeti hosts dont correctly report lldp_parent - https://phabricator.wikimedia.org/T289679 (10jbond) 05Resolved→03Open While rolling out the lldp factupdate i noticed an some machines have ip_forwarding enabled. thi... [11:56:46] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Modify homer/automation templates to support 100BaseTX interfaces with autoneg disabled. - https://phabricator.wikimedia.org/T288343 (10cmooney) Sorted it eventually :) ` cmooney@mr1-ulsfo> show interfaces ge-0/0/0... [11:57:00] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Modify homer/automation templates to support 100BaseTX interfaces with autoneg disabled. - https://phabricator.wikimedia.org/T288343 (10cmooney) 05Open→03Resolved [14:09:19] 10Traffic, 10SRE, 10SRE-swift-storage, 10Thumbor: Thumbnail of deleted image shown in "File history" after new image with same filename got uploaded - https://phabricator.wikimedia.org/T281780 (10jcrespo) I ran the following: ` $ mwscript purgeChangedFiles.php --wiki=commonswiki --starttime=2021051112254... [14:18:44] 10Traffic, 10SRE, 10SRE-swift-storage, 10Thumbor: Thumbnail of deleted image shown in "File history" after new image with same filename got uploaded - https://phabricator.wikimedia.org/T281780 (10jcrespo) I'm silly, the issue was on swift, not on cache: ` curl -I http://ms-fe.svc.eqiad.wmnet/wikipedia/com... [15:10:29] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE: (Need By: TBD) rack/setup/install atlas-codfw.wikimedia.org - https://phabricator.wikimedia.org/T273114 (10cmooney) Papaul rebooted yesterday with the USB key present. Things //appeared// to go ok, on the serial console the device went into a Linux... [15:41:35] 10netops, 10Infrastructure-Foundations, 10SRE: 2021-08-26 Primary inbound port utilisation over 80% page for mr1-esams.wikimedia.org - https://phabricator.wikimedia.org/T289820 (10jcrespo) Commenting as I think @ayounsi will not have been CCed on the original Phab report, for him to triage. [16:23:29] 10Traffic, 10DNS, 10SRE: DNS entries for WikiLearn dev servers - https://phabricator.wikimedia.org/T289618 (10Ijon) 05Resolved→03Open (I'm assuming it's easier to re-open this for further related requests. If that's not the case, let me know!) Can you please add the following CNAMEs to the learn.wiki DN... [16:41:53] 10netops, 10Infrastructure-Foundations, 10SRE: 2021-08-26 Primary inbound port utilisation over 80% page for mr1-esams.wikimedia.org - https://phabricator.wikimedia.org/T289820 (10cmooney) I had a look at this this morning (didn't catch the page when it fired and it cleared quickly as you say). Seems to be... [16:42:13] 10netops, 10Infrastructure-Foundations, 10SRE: 2021-08-26 Primary inbound port utilisation over 80% page for mr1-esams.wikimedia.org - https://phabricator.wikimedia.org/T289820 (10cmooney) p:05Triage→03Low [17:18:24] 10Traffic, 10DNS, 10SRE: DNS entries for WikiLearn dev servers - https://phabricator.wikimedia.org/T289618 (10Asaf) And also: stage.learn.wiki studio.stage.learn.wiki preview.stage.learn.wiki CNAME = http://wkm-stage-alb-1830818829.us-east-1.elb.amazonaws.com [17:40:43] 10Traffic, 10SRE, 10SRE-swift-storage, 10Thumbor: Thumbnail of deleted image shown in "File history" after new image with same filename got uploaded - https://phabricator.wikimedia.org/T281780 (10AntiCompositeNumber) fwiw this issue was recently reported on https://commons.wikimedia.org/wiki/File:Anshu_Dik... [18:59:47] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-eqiad, 10cloud-services-team (Hardware): (Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10RobH) [19:02:01] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-eqiad, 10cloud-services-team (Hardware): (Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10RobH) [19:03:50] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): CloudVPS: IPv6 early PoC - https://phabricator.wikimedia.org/T245495 (10faidon) [19:04:10] 10netops, 10Infrastructure-Foundations, 10SRE: Cloud IPv6 subnets - https://phabricator.wikimedia.org/T187929 (10faidon) 05Open→03Stalled There are some ongoing conversations with the WMCS team regarding the placement of their infrastructure in our network/infrastructure, and I think it would be good to... [19:05:29] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-eqiad, 10cloud-services-team (Hardware): Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10wiki_willy) [21:27:56] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp2027:9331 is unreachable - https://alerts.wikimedia.org [21:55:56] (VarnishTrafficDrop) firing: 66% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [22:15:56] (VarnishTrafficDrop) resolved: 67% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [23:09:36] 10Traffic, 10SRE, 10serviceops, 10Datacenter-Switchover: Services without a service IP cannot automatically be switched by the switchdc cookbook - https://phabricator.wikimedia.org/T285707 (10Legoktm)