[10:20:48] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10jijiki) [11:05:49] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10jijiki) [11:11:00] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10jijiki) [13:51:11] 10Traffic, 10Infrastructure-Foundations, 10SRE: Anycast: Add IPv6 support to bird and anycast-healthchecker (Puppet) - https://phabricator.wikimedia.org/T292737 (10ssingh) [14:01:43] 10Traffic, 10Infrastructure-Foundations, 10SRE: Anycast: Add IPv6 support to bird and anycast-healthchecker (Puppet) - https://phabricator.wikimedia.org/T292737 (10ssingh) [14:01:49] 10Traffic, 10SRE, 10Patch-For-Review: Deploy Wikidough: Experimental DNS-over-HTTPS (DoH) public resolver - https://phabricator.wikimedia.org/T252132 (10ssingh) [16:15:57] (VarnishTrafficDrop) firing: 69% GET drop in text@eqiad during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [16:25:56] (VarnishTrafficDrop) resolved: 69% GET drop in text@eqiad during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [17:08:50] 10Traffic, 10SRE, 10SRE Observability (FY2021/2022-Q2): Investigate cp5006 crash - https://phabricator.wikimedia.org/T292506 (10herron) FWIW I do see syslog entries had arrived to `centrallog1001:/srv/syslog/cp5006/syslog.log-2021100[34].gz` between Oct 2 17:36:22 and Oct 3 at 08:26:37 which is good news in... [17:51:20] 10netops, 10Infrastructure-Foundations, 10SRE: Eqiad Expansion - LVS Connectivity Options - https://phabricator.wikimedia.org/T292630 (10cmooney) @BBlack @mark @joanna_borun @ayounsi I've put together a doc outlining some options here. Purpose of the doc is to facilitate collaboration / discussion on the op... [18:38:53] 10Traffic, 10DNS, 10SRE: Additional DNS entries for Wikilearn project (Community Development) - https://phabricator.wikimedia.org/T292537 (10Ijon) 05Resolved→03Open Hi! The forum.learn.wiki CNAME is incorrect. We asked for: forum.learn.wiki wkm-prod-alb-30644061.us-east-1.elb.amazonaws.com but it l... [18:51:03] 10Traffic, 10DNS, 10SRE, 10Patch-For-Review: Additional DNS entries for Wikilearn project (Community Development) - https://phabricator.wikimedia.org/T292537 (10ssingh) 05Open→03Resolved >>! In T292537#7409931, @Ijon wrote: > Hi! The forum.learn.wiki CNAME is incorrect. We asked for: > > forum.learn... [18:57:57] (VarnishTrafficDrop) firing: 58% GET drop in text@eqsin during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [19:02:57] (VarnishTrafficDrop) resolved: 66% GET drop in text@eqsin during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [19:08:57] (VarnishTrafficDrop) firing: 43% GET drop in text@eqsin during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [19:13:57] (VarnishTrafficDrop) resolved: (2) 67% GET drop in text@eqsin during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [19:48:57] (VarnishTrafficDrop) firing: 69% GET drop in text@eqsin during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [19:53:57] (VarnishTrafficDrop) resolved: 69% GET drop in text@eqsin during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [20:18:49] 10Traffic, 10SRE: Wikipedia shutdown in Russia - https://phabricator.wikimedia.org/T292776 (10RhinosF1) Hi, A public task with information will be coming from SRE (@cdanis) soon but this was known. [20:57:33] 10Traffic, 10SRE: Wikipedia not accessible in Russia on 2021-10-07 16:00-17:00UTC - https://phabricator.wikimedia.org/T292776 (10Aklapper) [21:02:16] 10Traffic, 10SRE: Wikipedia not accessible in Russia on 2021-10-07 16:00-17:00UTC - https://phabricator.wikimedia.org/T292776 (10CDanis)