[10:13:46] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Can't commit on asw-b-codfw - https://phabricator.wikimedia.org/T295118 (10ayounsi) Current status: * IPv6 is still broken on asw-b7-codfw (for traffic local and transiting through the switch) * `inet6` is disabled on cr2-co... [10:40:29] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Can't commit on asw-b-codfw - https://phabricator.wikimedia.org/T295118 (10ayounsi) Hopefully we won't need to, but if asw1-b2-codfw needs to be rebooted, here are the impacted servers: ms-be2041 ms-be2046 ms-be2031 ms-be203... [14:16:23] cwhite: what would be the best way of delivering logs to rsyslog bypassing systemd's journald? [14:17:16] directly add a UNIX socket definition on rsyslog snippet? [14:17:34] $AddUnixListenSocket /run/haproxy-syslog.socket? [14:42:53] that solved the problem :) [14:48:58] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Can't commit on asw-b-codfw - https://phabricator.wikimedia.org/T295118 (10Vgutierrez) @ayounsi both lvs2008 and lvs2009 are primary LVS, so lvs2010 would assume the load of both during asw1-b2-codfw reboot. Far from ideal b... [15:03:58] 10Traffic, 10Observability-Metrics, 10SRE, 10Patch-For-Review, 10User-ema: varnishmtail metric loss due to mtail not reading from pipe fast enough - https://phabricator.wikimedia.org/T293879 (10ema) The situation has improved significantly, we are now processing [[ https://grafana.wikimedia.org/d/wiU3SdE... [15:34:49] glad it worked! :) [21:52:05] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10SRE, 10Beta-Cluster-reproducible: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021. - https://phabricator.wikimedia.org/T296113 (10AlexisJazz) [21:52:36] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10Quality-and-Test-Engineering-Team (QTE), and 2 others: [epic] The SSL certificate for Beta cluster domains fails to properly renew & deploy - https://phabricator.wikimedia.org/T293585 (10AlexisJazz) [21:52:42] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10SRE, 10Beta-Cluster-reproducible: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021. - https://phabricator.wikimedia.org/T296113 (10AlexisJazz) [21:52:55] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10SRE, 10Beta-Cluster-reproducible: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021. - https://phabricator.wikimedia.org/T296113 (10RhinosF1) [21:53:36] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10SRE, 10Beta-Cluster-reproducible: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021. - https://phabricator.wikimedia.org/T296113 (10RhinosF1) [21:53:42] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10Quality-and-Test-Engineering-Team (QTE), and 2 others: [epic] The SSL certificate for Beta cluster domains fails to properly renew & deploy - https://phabricator.wikimedia.org/T293585 (10RhinosF1) [21:55:08] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10SRE, 10Beta-Cluster-reproducible: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021. - https://phabricator.wikimedia.org/T296113 (10RhinosF1) Closing as duplicate of linked task. I assume it's the standard restart of st... [21:55:21] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10SRE, 10Beta-Cluster-reproducible: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021. - https://phabricator.wikimedia.org/T296113 (10AlexisJazz) @RhinosF1 : en.wikipedia.beta.wmflabs.org is already working again but uplo... [21:56:18] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10SRE, 10Beta-Cluster-reproducible: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021. - https://phabricator.wikimedia.org/T296113 (10RhinosF1) >>! In T296113#7517656, @AlexisJazz wrote: > @RhinosF1 : en.wikipedia.beta.wm... [21:58:22] 10HTTPS, 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021. - https://phabricator.wikimedia.org/T296113 (10RhinosF1) Removing SRE + Traffic as it's not managed by them [21:58:29] 10HTTPS, 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021. - https://phabricator.wikimedia.org/T296113 (10AlexisJazz) >>! In T296113#7517658, @RhinosF1 wrote: >>>! In T296113#7517656, @AlexisJazz wrote: >> @RhinosF1... [21:59:30] 10HTTPS, 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021. - https://phabricator.wikimedia.org/T296113 (10RhinosF1) Upload hosts are just more broken than the rest. Yes I'm 1000% sure it's the exact same issue and i... [22:01:44] 10HTTPS, 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021. - https://phabricator.wikimedia.org/T296113 (10AlexisJazz) >>! In T296113#7517666, @RhinosF1 wrote: > Upload hosts are just more broken than the rest. Yes I... [22:03:06] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10Quality-and-Test-Engineering-Team (QTE), and 2 others: [epic] The SSL certificate for Beta cluster domains fails to properly renew & deploy - https://phabricator.wikimedia.org/T293585 (10Urbanecm) [22:06:26] 10HTTPS, 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021. - https://phabricator.wikimedia.org/T296113 (10RhinosF1) Yes I've noticed that but it still impacted the whole of beta. Someone can dig on the task from yes... [22:11:25] 10HTTPS, 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021. - https://phabricator.wikimedia.org/T296113 (10AlexisJazz) >>! In T296113#7517683, @RhinosF1 wrote: > Yes I've noticed that but it still impacted the whole... [22:12:27] 10HTTPS, 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021. - https://phabricator.wikimedia.org/T296113 (10RhinosF1) It's the same certificate for all. It's just that upload requires an extra step to deploy because f... [23:16:56] (EdgeTrafficDrop) firing: 54% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org [23:21:57] (EdgeTrafficDrop) resolved: 66% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org [23:39:56] (EdgeTrafficDrop) firing: 55% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org [23:49:56] (EdgeTrafficDrop) resolved: 65% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org