[06:47:56] (EdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [07:02:56] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [09:00:08] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: Q3:(Need By: ASAP) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10ayounsi) a:05ayounsi→03RobH What's left to do on the network side: ` 1/ cr1-drmrs:et-0/0/2 currently connected to: asw1-b12-drmrs:et... [10:50:18] 10Traffic, 10DNS, 10SRE, 10Patch-For-Review, 10WMSE (IT): Need Assistance adding DNS records to claim domain - https://phabricator.wikimedia.org/T300076 (10jbond) This is now in p[lace, please re-open if you still see issues ` lang=shell $ dig txt wikimedia.org @ns0.wikimedia.org... [10:58:41] Anyone able to responde to my questions from yesterday? and in genral what is the best way to request a review from the traffic team? [10:59:01] question from yesterday "hi all i created a PS in december after a DDoS from a cloud provider where it was [10:59:04] request that we have some genral rate limits on cloud provideres. i have created a [10:59:07] very genral patch for that [10:59:10] https://gerrit.wikimedia.org/r/c/operations/puppet/+/740818 which just needs someone [10:59:13] to comment on what reasnable limits would be. (have also cross posted this to [10:59:16] serviceops) [10:59:18] there is also a more generic implmentation which alows use to change theses limits [10:59:21] via hiera https://gerrit.wikimedia.org/r/c/operations/puppet/+/740828/11 [10:59:24] (both looking for review)" [11:28:59] jbond: sorry about not answering yesterday.. my brain raised a timeout and returned E_TOOBUSY [11:29:41] pinging us here should be enough, maybe mention my nick and/or b.black to get sure that we get pinged [11:31:03] vgutierrez: ack thanks and no problem we all get busy :), but yes i wasn't sure if i should ping directly or perhaps ping you directly in a different channle like sre. thanks will one or all of you next time :) [11:36:26] please feel free to ping Kwaku also, new manager of the team [11:37:16] jbond: yeah.. also I'm used to delegate the varnish stuff to e.m.a but that doesn't seem possibly anymore 😓 [11:37:23] *possible [11:38:27] thank mark, vgutierrez yes that was something else i wasn't sure on, i would normally just ping ema directly for vcl stuff in there absenses i wasn't sure who best to bug [11:39:06] during EU time you'll be trapped with my limited capacities ;P [11:39:25] im sure that will be more then enough :) [12:12:01] 10Traffic, 10SRE, 10WMF-Legal, 10Performance-Team (Radar), 10Privacy: Consider disabling Chrome Lite pages for Wikipedia on Chrome on mobile with Cache-Control: no-transform - https://phabricator.wikimedia.org/T218618 (10Peter) This is also interesting, the coming Chrome prefetch proxy on Android is roll... [18:46:14] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: Q3:(Need By: ASAP) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10RobH) CS0451899 > Support, > > We recently filed CS0334187 for the installation of two of our routers. During that installation, it...