[00:39:56] (EdgeTrafficDrop) firing: 57% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org [00:54:56] (EdgeTrafficDrop) resolved: 68% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org [06:51:56] (EdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [07:06:56] (EdgeTrafficDrop) resolved: 67% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [09:59:55] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Enable IPv6 for Wikidough - https://phabricator.wikimedia.org/T301165 (10cmooney) 05In progress→03Resolved [10:00:23] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Enable IPv6 for Wikidough - https://phabricator.wikimedia.org/T301165 (10cmooney) [10:00:54] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Enable IPv6 for Wikidough - https://phabricator.wikimedia.org/T301165 (10cmooney) Ok gonna close this one, range announced and doh working on IPv6 from all our POPs now. I've a separate task - T301900 - to validate the route p... [10:38:14] 10Traffic, 10SRE, 10Patch-For-Review: Deploy Wikidough: Experimental DNS-over-HTTPS (DoH) public resolver - https://phabricator.wikimedia.org/T252132 (10ssingh) [10:38:19] 10Traffic, 10Infrastructure-Foundations, 10SRE: Anycast: Add IPv6 support to bird and anycast-healthchecker (Puppet) - https://phabricator.wikimedia.org/T292737 (10ssingh) 05Open→03Resolved IPv6 support for Wikidough and durum was finalized in T301165. Thanks to Arzhel, Cathal, and John Bond for all the... [11:04:12] hi folks [11:04:50] in https://phabricator.wikimedia.org/T300164#7717677 I proposed to drop the varnishkafka package in the main components of our apt repos, to avoid any issue due to wrong versioning in the future [11:05:12] it seems harmful to have both versions, but lemme know your thoughts [11:06:13] (the idea is to keep a single canonical version of varnishkafka at any given time, unless we are upgrading or something) [11:06:25] 10Traffic, 10DNS, 10SRE, 10Patch-For-Review, 10WMSE (IT): Need Assistance adding DNS records to claim domain - https://phabricator.wikimedia.org/T300076 (10jbond) this is in place now, notice the list TXT line below ` lang=console $ dig txt wikimedia.org @ns0.wikimedia.org... [11:06:45] we could also move the 1.1.0 version to main, and remove it from the varnish6 component [11:06:48] as you folks prefer [11:11:09] 10Traffic, 10DNS, 10SRE, 10WMSE (IT): Need Assistance adding DNS records to claim domain - https://phabricator.wikimedia.org/T300076 (10jbond) 05Open→03Stalled [11:40:41] hi all i created a PS in december after a DDoS from a cloud provider where it was request that we have some genral rate limits on cloud provideres. i have created a very genral patch for that https://gerrit.wikimedia.org/r/c/operations/puppet/+/740818 which just needs someone to comment on what reasnable limits would be. (have also cross posted this to serviceops) [11:41:16] there is also a more generic implmentation which alows use to change theses limits via hiera https://gerrit.wikimedia.org/r/c/operations/puppet/+/740828/11 [11:42:27] (both looking for review) [12:43:26] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10jbond) i ran a script yesterday which has collected all the current drac and bios versions. Sorry ab... [15:25:46] bblack,vgutierrez - hi :) when you have a moment, what do you think about removing varnishkafka from the main component of {stretch,buster}-wikimedia? In this way we'll have only the right vk version on the varnish6 component (see above for more info, I wrote earlier on with some links) [15:41:19] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure: The certificate for upload.wikimedia.beta.wmflabs.org expired on February 16, 2022. - https://phabricator.wikimedia.org/T301995 (10AlexisJazz) [15:41:51] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure: The certificate for upload.wikimedia.beta.wmflabs.org expired on February 16, 2022. - https://phabricator.wikimedia.org/T301995 (10AlexisJazz) [15:41:54] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10Quality-and-Test-Engineering-Team (QTE), and 2 others: [epic] The SSL certificate for Beta cluster domains fails to properly renew & deploy - https://phabricator.wikimedia.org/T293585 (10AlexisJazz) [16:05:06] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10SRE: The certificate for upload.wikimedia.beta.wmflabs.org expired on February 16, 2022. - https://phabricator.wikimedia.org/T301995 (10Zabe) Since there seems to be a valid, I did the same mitigation as in T271808 and T293070. ` root@deployment-cache-up... [16:06:32] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10SRE: The certificate for upload.wikimedia.beta.wmflabs.org expired on February 16, 2022. - https://phabricator.wikimedia.org/T301995 (10AlexisJazz) >>! In T301995#7718593, @Zabe wrote: > Since there seems to be a valid certificate, I did the same mitigat... [16:55:02] elukey: so I've mentioned to mmandere a few weeks ago that we should move varnish6 and friends to the main component and remove the varnish6 component cause it doesn't make any sense to keep varnish5 on the main component and varnish6 on its component nowadays [16:55:40] yep good for me [17:40:53] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: ASAP) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10ayounsi) Current status: * Physical work left (I'll give the details tomorrow): ** Planned: move Telia's link to the routers now that we h... [17:51:32] mmandere: o/ when you have a time let's do the cleanup above, so we avoid risking to get weird varnishkafka packages [18:33:02] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: Q3:(Need By: ASAP) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10RobH) I can put in a followup ticket for them to correct the 'unplanned' items but I'll wait until you finish your setup or give the go a... [18:33:10] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: Q3:(Need By: ASAP) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10RobH) a:05RobH→03ayounsi [18:35:08] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: Q3:(Need By: ASAP) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10RobH) No packing slips in box according to the Interxion engineer who did our remote hands work, so I've requested a copy from Myriad so... [23:07:28] 10Traffic, 10DNS, 10SRE, 10WMSE (IT): Need Assistance adding DNS records to claim domain - https://phabricator.wikimedia.org/T300076 (10MRamirez_WMF) @jbond I have the records to finalize the transition TXT name Copy record‎@‎ (or skip if not supported by provider) TXT value MS=ms70322281 TTL ‎3600‎ (or y...