[07:45:48] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Data-Platform-SRE (2024.05.06 - 2024.05.26): an-worker1165.eqiad.wmnet and increased network activity resulting in page on May 13 2024 - https://phabricator.wikimedia.org/T364893#9798455 (10Gehel)
[10:52:57] <wikibugs>	 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 06SRE: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184#9799115 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1041.eqiad.wmnet with OS bookworm
[11:05:53] <wikibugs>	 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184#9799155 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1041.eqiad.wmnet with...
[11:11:11] <wikibugs>	 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184#9799171 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1041.eqiad.wmnet...
[11:52:49] <wikibugs>	 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184#9799293 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1041.eqiad.wmnet with...
[15:28:21] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Data-Platform-SRE (2024.05.06 - 2024.05.26): an-worker1165.eqiad.wmnet and increased network activity resulting in page on May 13 2024 - https://phabricator.wikimedia.org/T364893#9800676 (10BTullis) I think that the most likely candidate at the moment is user-gen...
[15:28:38] <jinxer-wm>	 FIRING: LVSRealserverMSS: Unexpected MSS value on 198.35.26.112:443 @ cp4050 - TODO - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS
[15:31:31] <vgutierrez>	 ^^ checking
[15:33:38] <jinxer-wm>	 RESOLVED: LVSRealserverMSS: Unexpected MSS value on 198.35.26.112:443 @ cp4050 - TODO - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS
[15:34:44] <sukhe>	 vgutierrez: what was the resolution?
[15:35:12] <vgutierrez>	 false positive :|
[15:36:17] <sukhe>	 oh
[16:07:38] <jinxer-wm>	 FIRING: [4x] LVSRealserverMSS: Unexpected MSS value on [2620:0:863:ed1a::2:b]:443 @ cp4045 - TODO - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS
[16:12:38] <jinxer-wm>	 RESOLVED: [4x] LVSRealserverMSS: Unexpected MSS value on [2620:0:863:ed1a::2:b]:443 @ cp4045 - TODO - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS
[16:17:08] <jinxer-wm>	 FIRING: [7x] LVSRealserverMSS: Unexpected MSS value on [2620:0:863:ed1a::2:b]:443 @ cp4045 - TODO - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS
[16:18:01] <sukhe>	 ^ vgutierrez false positives again?
[16:22:08] <jinxer-wm>	 RESOLVED: [7x] LVSRealserverMSS: Unexpected MSS value on [2620:0:863:ed1a::2:b]:443 @ cp4045 - TODO - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS
[16:27:08] <jinxer-wm>	 FIRING: [7x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.112:80 @ cp4050 - TODO - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS
[16:27:53] <jinxer-wm>	 FIRING: [11x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.112:80 @ cp4050 - TODO - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS
[16:28:01] <vgutierrez>	 I think so
[16:29:56] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Data-Platform-SRE (2024.05.06 - 2024.05.26): an-worker1165.eqiad.wmnet and increased network activity resulting in page on May 13 2024 - https://phabricator.wikimedia.org/T364893#9801014 (10cmooney) >>! In T364893#9800673, @BTullis wrote: > I think that the most...
[16:32:08] <jinxer-wm>	 FIRING: [7x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.112:80 @ cp4050 - TODO - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS
[16:32:53] <jinxer-wm>	 FIRING: [13x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.112:443 @ cp4049 - TODO - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS
[16:33:29] <sukhe>	 ^ above alert silenced (based on vg's recommendation)
[16:37:08] <jinxer-wm>	 RESOLVED: [9x] LVSRealserverMSS: Unexpected MSS value on [2620:0:863:ed1a::2:b]:443 @ cp4048 - TODO - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS
[16:37:53] <jinxer-wm>	 FIRING: [11x] LVSRealserverMSS: Unexpected MSS value on [2620:0:863:ed1a::2:b]:443 @ cp4046 - TODO - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS
[16:42:53] <jinxer-wm>	 FIRING: [13x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.112:443 @ cp4047 - TODO - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS
[16:47:08] <jinxer-wm>	 FIRING: [13x] LVSRealserverMSS: Unexpected MSS value on 198.35.26.112:443 @ cp4047 - TODO - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=2&var-site=ulsfo&var-cluster=cache_upload - https://alerts.wikimedia.org/?q=alertname%3DLVSRealserverMSS
[22:41:31] <wikibugs>	 06Traffic, 06Movement-Insights: Disable Chrome Private Prefetch Proxy - https://phabricator.wikimedia.org/T364126#9802385 (10Mayakp.wiki) [[ https://docs.google.com/document/d/1BUvOuSsHPzQbbpBhZZN4761oCU_kI1qOVEE1q-3fz-A/edit#heading=h.xq5iwvrhfvrv | Disable Chrome Private Prefetch Proxy - Monitoring Plan ]] l...