[06:49:56] (EdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [07:04:56] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [08:30:14] 10netops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10hashar) [08:31:35] 10netops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10hashar) >>! In T283582#7606038, @Dzahn wrote: > for the record: I have absolutel... [10:14:34] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10ayounsi) a quick look on GitHub shows 2 approaches: * This one parses the firmware page: https://gith... [11:49:53] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10jbond) > This one parses the firmware page: https://github.com/lateralblast/druid This uses selenium... [11:58:40] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10MoritzMuehlenhoff) >>! In T283771#7608623, @jbond wrote: >> This one parses the firmware page: https:... [12:02:12] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10jbond) > It'll also be rather brittle since every year Firefox ESR bumps to a new version (e.g. from... [12:13:06] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10ayounsi) Relying on parsing a website is often asking for troubles. Maybe we can also ask our account... [12:18:34] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10MoritzMuehlenhoff) >>! In T283771#7608655, @jbond wrote: >> It'll also be rather brittle since every... [12:21:20] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10jbond) > Maybe we can also ask our account rep. for their recommendation (different API, etc). willy... [14:04:18] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10jbond) I had another look at the dell pages and i have worked out a way to pull via an undocumented a... [18:06:07] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: cr-codfw: set up static route for 185.15.57.8/30 - https://phabricator.wikimedia.org/T295288 (10Krinkle) [19:48:03] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar), 10User-ema: Package and deploy Varnish 6.0.9 - https://phabricator.wikimedia.org/T298758 (10kchapman) [19:56:57] 10netops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10SRE, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10Dzahn) ACK, understood @Hashar, then this goes back to #ops-codfw [19:58:56] 10netops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10SRE, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10Dzahn) @Papaul it seems that "broken DRAC" is actually the reason for b... [20:00:26] 10netops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10SRE, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10Dzahn)