[06:41:56] <jinxer-wm>	 (EdgeTrafficDrop) firing: 66% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org
[06:46:56] <jinxer-wm>	 (EdgeTrafficDrop) resolved: 66% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org
[06:49:56] <jinxer-wm>	 (EdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org
[06:54:56] <jinxer-wm>	 (EdgeTrafficDrop) resolved: (2) 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org
[06:56:26] <jinxer-wm>	 (EdgeTrafficDrop) firing: (2) 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org
[07:01:26] <jinxer-wm>	 (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org
[10:04:38] <btullis>	 Morning. I'm hoping to get a review of an LVS change please. https://gerrit.wikimedia.org/r/c/operations/puppet/+/769398
[10:04:38] <btullis>	 I'm proceeding cautiously because I haven't gone through this procedure before.
[12:00:47] <vgutierrez>	 btullis: you need to set a proper weight for your servers, 0 isn't a valid one
[12:01:00] <vgutierrez>	 {"datahubsearch1001.eqiad.wmnet": {"weight": 0, "pooled": "yes"}, "tags": "dc=eqiad,cluster=datahubsearch,service=opensearch"}
[12:14:33] <btullis>	 vgutierrez: OK, thanks. Is it OK if I set them all to 10 ?
[12:14:41] <vgutierrez>	 it is
[12:17:07] <btullis>	 Done, thanks.
[12:30:05] <vgutierrez>	 btullis: as mentioned on the CR, you can go ahead, please note that the involved LVS are lvs1020 (secondary) and lvs1019 (primary for low-traffic)
[13:43:02] <btullis>	 vgutierrez: Many thanks. Proceeding now'ish.
[13:52:06] <vgutierrez>	 https://www.irccloud.com/pastebin/m1qAMEt8/
[13:52:11] <vgutierrez>	 looking good on lvs1020
[13:52:37] <vgutierrez>	 same from ipvs PoV
[13:52:41] <vgutierrez>	 https://www.irccloud.com/pastebin/HEMoDoOb/
[14:13:46] <btullis>	 https://www.irccloud.com/pastebin/m5lZfhhA/
[14:14:09] <btullis>	 Looks OK to me, but I'm not getting through to the backend server yet.
[14:43:20] <btullis>	 Figured out why. The backend opensearch isn't listening on the realserver_ip, only on its primary IP. Will fix that now.
[14:55:43] <wikibugs>	 10Traffic, 10User-Ladsgroup: Rework education.wikimedia.org redirects - https://phabricator.wikimedia.org/T303397 (10Ladsgroup)
[18:15:06] <wikibugs>	 10Traffic, 10SRE, 10envoy, 10serviceops, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10RLazarus)
[20:44:51] <wikibugs>	 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10nskaggs)
[20:46:46] <wikibugs>	 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10nskaggs) @RobH I updated the task to call out these should be installed into two different rows, as well as not installing in WMCS...
[20:48:57] <wikibugs>	 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10nskaggs) Note, the existing machines are taking up a total of 12U (6U each) in D2 and A4.