[08:30:14] 10netops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10hashar) [08:31:35] 10netops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10hashar) >>! In T283582#7606038, @Dzahn wrote: > for the record: I have absolutel... [09:56:20] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Unused puppet resources audit, 2021 - https://phabricator.wikimedia.org/T272559 (10dcaro) [10:14:34] 10SRE-tools, 10netops, 10DC-Ops, 10Infrastructure-Foundations, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10ayounsi) a quick look on GitHub shows 2 approaches: * This one parses the firmware page: https:... [11:49:53] 10SRE-tools, 10netops, 10DC-Ops, 10Infrastructure-Foundations, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10jbond) > This one parses the firmware page: https://github.com/lateralblast/druid This uses sel... [11:58:40] 10SRE-tools, 10netops, 10DC-Ops, 10Infrastructure-Foundations, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10MoritzMuehlenhoff) >>! In T283771#7608623, @jbond wrote: >> This one parses the firmware page:... [12:02:11] 10SRE-tools, 10netops, 10DC-Ops, 10Infrastructure-Foundations, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10jbond) > It'll also be rather brittle since every year Firefox ESR bumps to a new version (e.g.... [12:13:06] 10SRE-tools, 10netops, 10DC-Ops, 10Infrastructure-Foundations, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10ayounsi) Relying on parsing a website is often asking for troubles. Maybe we can also ask our a... [12:18:34] 10SRE-tools, 10netops, 10DC-Ops, 10Infrastructure-Foundations, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10MoritzMuehlenhoff) >>! In T283771#7608655, @jbond wrote: >> It'll also be rather brittle since... [12:19:28] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Unused puppet resources audit, 2021 - https://phabricator.wikimedia.org/T272559 (10dcaro) [12:21:20] 10SRE-tools, 10netops, 10DC-Ops, 10Infrastructure-Foundations, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10jbond) > Maybe we can also ask our account rep. for their recommendation (different API, etc).... [13:35:06] I have a dr apt right before our meeting, depending on the waiting time I might be late to our meeting [14:04:17] 10SRE-tools, 10netops, 10DC-Ops, 10Infrastructure-Foundations, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10jbond) I had another look at the dell pages and i have worked out a way to pull via an undocume... [15:05:47] jhathaway: topranks: are you avalible for the meeting? [15:06:04] were chatting away here! [15:06:12] topranks and I were chatting with each other! [15:06:12] do we have the wrong link? [15:06:22] https://meet.google.com/chw-ydxp-ddo?authuser=0 [15:06:38] this is the link i have [15:48:42] jbond, topranks, looks like they have this data on their API: https://www.teliacarrier.com/portal/developer?tab=documentation [15:49:43] nice! [15:49:45] see the "/security-ddos/mitigation-reports" [15:49:55] yeah just reading that one [15:50:05] I'm wondering what else could useful to query and graph [15:50:33] and if we can have private grafana dashboards [15:51:55] hmm yeah, like periodically poll their api and push the data into Prometheus metrics we can then graph? [15:52:22] XioNoX: where are the credentials for the potal? [15:52:50] let me invite you [15:53:42] great thanks [15:54:02] jbond: you should have an email [15:54:21] XioNoX: yes got it thanks [15:54:44] topranks: yeah, it could display the mitigated ddos, maybe graph the PPS/Gbps [15:54:56] not a bad idea at all [15:58:16] yeah, their new dashboard is pretty cool: https://www.teliacarrier.com/portal/security-ddos/alerts/13420368?tab=traffic [15:58:34] and I bet it's all data from their API [15:59:54] +1 level of data is very good and visualizations too [16:00:31] Pity they can't scale the units (always shows "bytes", not KB/MB/GB) on the graphs though. [16:05:05] indeed [16:29:02] 10SRE-tools, 10Discovery, 10Infrastructure-Foundations, 10Discovery-Search (Current work), 10IPv6: Some Search Platform / Discovery clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271143 (10MPhamWMF) [16:54:35] 10SRE-tools, 10Discovery, 10Infrastructure-Foundations, 10Discovery-Search (Current work), 10IPv6: Some Search Platform / Discovery clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271143 (10MPhamWMF) [16:59:59] 10SRE-tools, 10Discovery, 10Infrastructure-Foundations, 10Discovery-Search (Current work), 10IPv6: Some Search Platform / Discovery clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271143 (10MPhamWMF) [18:06:07] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: cr-codfw: set up static route for 185.15.57.8/30 - https://phabricator.wikimedia.org/T295288 (10Krinkle) [19:56:57] 10netops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10SRE, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10Dzahn) ACK, understood @Hashar, then this goes back to #ops-codfw [19:58:56] 10netops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10SRE, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10Dzahn) @Papaul it seems that "broken DRAC" is actually the reason for b... [20:00:24] 10netops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10SRE, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10Dzahn) [20:20:50] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Unused puppet resources audit, 2021 - https://phabricator.wikimedia.org/T272559 (10Dzahn) [20:21:07] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Unused puppet resources audit, 2021 - https://phabricator.wikimedia.org/T272559 (10Dzahn) [23:47:20] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Znuny, 10fundraising-tech-ops: move donation,donate, donations (otrs, wikimania) exim aliases from SRE to ITS - https://phabricator.wikimedia.org/T297915 (10elappen-WMF) Commenting on behalf of the staff working on Wikimania to note that we'd like to move wi...