[06:00:56] (VarnishTrafficDrop) firing: 59% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [06:05:56] (VarnishTrafficDrop) resolved: 66% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [09:13:56] (VarnishTrafficDrop) firing: 15% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [09:18:56] (VarnishTrafficDrop) resolved: (2) 41% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [10:07:17] 10Traffic, 10Infrastructure-Foundations, 10Security, 10WMF-NDA: openssl: DSA-4963-1 - https://phabricator.wikimedia.org/T289671 (10jbond) [10:12:54] 10Traffic, 10Infrastructure-Foundations, 10Security, 10WMF-NDA: openssl: DSA-4963-1 - https://phabricator.wikimedia.org/T289671 (10jbond) bullseye patch lgtm [13:07:49] 10netops, 10Infrastructure-Foundations, 10Puppet, 10User-jbond: LLDP: Ganeti hosts dont correctly report lldp_parent - https://phabricator.wikimedia.org/T289679 (10jbond) p:05Triage→03Medium [14:02:35] 10Traffic, 10DNS, 10SRE: DNS entries for WikiLearn dev servers - https://phabricator.wikimedia.org/T289618 (10Asaf) [14:36:02] 10Traffic, 10MediaWiki-Uploading, 10SRE, 10serviceops, 10Wikimedia-production-error: Unexpected upload speed to commons - https://phabricator.wikimedia.org/T288481 (10Majavah) [15:04:57] FYI I was auditing flows codfw/eqiad and came across unexpected (to me) at least plaintext kafka cp2* -> kafka-main1* https://phabricator.wikimedia.org/T286038#7308561 [15:05:21] uh? [15:06:46] I haven't dug deeper but it is definitely there, at least the connection [15:07:52] could it be the daemon that handles the PURGE actions to take from eventgate? [15:08:26] probably [15:08:54] /etc/varnishkafka/ config have 9093 and not 9092 anywhere [15:09:01] (just checked) [15:10:10] looks like varnishkafka for statsd [15:10:14] statsV [15:10:21] varnishka 8655 root 13u IPv4 1576313486 0t0 TCP cp2039.codfw.wmnet:27263->kafka-main1002.eqiad.wmnet:9092 (ESTABLISHED) [15:10:25] root 8655 1.6 0.0 619496 98524 ? Ssl May24 2256:24 /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf [15:11:33] happy to open a task for tracking, unless there's one already ? [15:12:40] ahhh yes that makes sense, statsv might not have been moved to 9093 [15:12:57] right.. port isn't specified there and hence the default one (9092) is being used [15:13:07] should be very easy to move to 9093 [15:14:16] I'll submit a CR soon(TM) [15:14:31] thanks for bringing it to our attention godog <3 beer++ [15:14:54] * godog increases beer counter [15:15:03] vgutierrez: sure no problem! glad it is easy to fix [15:15:25] please double check the task too in case some traffic jumps out to you as unexpected [15:15:37] and now you've jinxed it 😅 [15:15:56] * godog looks at jinxer-wm [15:16:11] it was the bot! [15:16:54] bbiab [15:16:56] statsv doesn't send any PII data IIRC for the record (I understand though the value of adding TLS, just wanted to state that we haven't risked leaked sensitive info) [15:28:58] 10Traffic, 10SRE, 10vm-requests: Please create a Ganeti VM for durum in eqiad - https://phabricator.wikimedia.org/T289693 (10ssingh) [15:39:20] godog, elukey, mmandere: could I get a sanity check on https://gerrit.wikimedia.org/r/714795 && https://gerrit.wikimedia.org/r/c/operations/puppet/+/714796? [15:42:58] vgutierrez: sure, checking [15:44:34] idea is to check that everything is ok on cp4032 and after that apply the change globally [15:44:50] looking good yeah [15:45:10] I'll merge it tomorrow morning, kinda late already :) [15:45:20] thanks <3 [15:45:32] reviewed as well, all good [15:45:39] +1 [16:09:04] 10Traffic, 10SRE, 10vm-requests: Please create a Ganeti VM for durum in eqiad - https://phabricator.wikimedia.org/T289693 (10jcrespo) Hey, @ssingh I've seen you have created this task. Recently there was a discussion about [[ https://wikitech.wikimedia.org/w/index.php?title=Phabricator&diff=1892983&oldid=18... [16:22:49] 10Traffic, 10SRE, 10vm-requests: Please create a Ganeti VM for Wikidough in ulsfo - https://phabricator.wikimedia.org/T284349 (10jcrespo) This seems resolved based on T284349#7144169 ? [16:24:32] 10Traffic, 10SRE, 10vm-requests: Please create a Ganeti VM for Wikidough in ulsfo - https://phabricator.wikimedia.org/T284349 (10ssingh) 05Open→03Resolved Yes please, this is resolved. Thanks! [16:29:27] 10Traffic, 10SRE, 10vm-requests: Please create a Ganeti VM for durum in eqiad - https://phabricator.wikimedia.org/T289693 (10ssingh) >>! In T289693#7308865, @jcrespo wrote: > Hey, > > @ssingh I've seen you have created this task. Recently there was a discussion about [[ https://wikitech.wikimedia.org/w/inde... [16:32:45] 10Traffic, 10SRE, 10vm-requests: Please create a Ganeti VM for durum in eqiad - https://phabricator.wikimedia.org/T289693 (10jcrespo) No protocol breaking- I just asked in case you needed help, as I happen to be on clinic duty this week. I am also offering my help to do this together, I am not @Dzahn, but I... [16:33:02] 10Traffic, 10SRE, 10vm-requests: Please create a Ganeti VM for durum in eqiad - https://phabricator.wikimedia.org/T289693 (10jcrespo) p:05Triage→03High [16:57:35] 10Traffic, 10SRE, 10vm-requests: Please create a Ganeti VM for durum in eqiad - https://phabricator.wikimedia.org/T289693 (10jcrespo) I talked to @ssingh, he told me he planned to ask for review to @Dzahn so I will leave it to you two :-). [17:31:06] 10Traffic, 10DC-Ops, 10SRE, 10decommission-hardware, 10ops-ulsfo: decommission bast4002.wikimedia.org - https://phabricator.wikimedia.org/T288579 (10wiki_willy) a:05Jclark-ctr→03RobH Hi @ssingh - just a heads up to add "ops-ulsfo" as a project tag, when this is ready for dc-ops to unrack. Thanks, Willy [17:38:16] 10Traffic, 10DNS, 10SRE: DNS entries for WikiLearn dev servers - https://phabricator.wikimedia.org/T289618 (10jcrespo) Adding @Brandon @vgutierrez as sometimes notifications on newly created tickets are a bit unreliable. I am on clinic duty this week, so trivial patches is something I am happy to help with,... [17:48:13] 10Traffic, 10DC-Ops, 10SRE, 10decommission-hardware, 10ops-ulsfo: decommission bast4002.wikimedia.org - https://phabricator.wikimedia.org/T288579 (10RobH) [17:53:40] 10Traffic, 10DC-Ops, 10SRE, 10decommission-hardware, 10ops-eqiad: decommission cescout1001.eqiad.wmnet - https://phabricator.wikimedia.org/T275696 (10wiki_willy) a:05Jclark-ctr→03Cmjohnson Hi @ssingh - just a heads up to add the "ops-eqiad" project tag, when its ready for the dc-ops steps. Thanks, W... [17:56:32] 10Traffic, 10DC-Ops, 10SRE, 10decommission-hardware, 10ops-eqiad: decommission cescout1001.eqiad.wmnet - https://phabricator.wikimedia.org/T275696 (10ssingh) Thanks, Willy! I will make sure to do it in the future. [17:59:23] 10Traffic, 10DC-Ops, 10SRE, 10decommission-hardware, 10ops-eqiad: reclaim cescout1001.eqiad.wmnet - https://phabricator.wikimedia.org/T275696 (10Cmjohnson) [18:00:10] 10Traffic, 10DC-Ops, 10SRE, 10decommission-hardware, 10ops-ulsfo: decommission bast4002.wikimedia.org - https://phabricator.wikimedia.org/T288579 (10RobH) So I'm not sure if we want to power this off and not use it at all, or re-allocate it as another service/host in ulsfo. My first thought was potentia... [18:03:44] 10Traffic, 10DC-Ops, 10SRE, 10decommission-hardware, 10ops-ulsfo: decommission bast4002.wikimedia.org - https://phabricator.wikimedia.org/T288579 (10RobH) IRC Update from my chat with @BBlack This old host is non-ideal but would work as a fallback ganeti host. I'll create a setup task for the host to r... [18:13:00] 10Traffic, 10DC-Ops, 10ops-ulsfo: (Need By: TBD) rack/setup/install ganeti4004 - https://phabricator.wikimedia.org/T289715 (10RobH) [18:13:14] 10Traffic, 10DC-Ops, 10ops-ulsfo: (Need By: TBD) rack/setup/install ganeti4004 - https://phabricator.wikimedia.org/T289715 (10RobH) [18:13:20] 10Traffic, 10DC-Ops, 10SRE, 10decommission-hardware, 10ops-ulsfo: decommission bast4002.wikimedia.org - https://phabricator.wikimedia.org/T288579 (10RobH) [18:16:03] 10Traffic, 10DC-Ops, 10SRE, 10decommission-hardware, 10ops-ulsfo: decommission bast4002.wikimedia.org - https://phabricator.wikimedia.org/T288579 (10RobH) 05Open→03Resolved [18:33:07] 10Traffic, 10DC-Ops, 10SRE, 10decommission-hardware, 10ops-eqiad: reclaim cescout1001.eqiad.wmnet - https://phabricator.wikimedia.org/T275696 (10wiki_willy) Much appreciated @ssingh, thanks! [18:37:51] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Three ports on asw2-d-eqiad are not working as expected - https://phabricator.wikimedia.org/T247881 (10Cmjohnson) @ayounsi I did some investigating on this today and there have been servers plugged into 2 of 3 (ge-1/0/5 and 1/0/6) ports now for qu... [18:48:19] 10Traffic, 10DNS, 10SRE, 10WMF-Communications, and 3 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776 (10Varnent) [18:58:27] 10Traffic, 10SRE-swift-storage, 10Thumbor: Thumbnail of deleted image shown in "File history" after new image with same filename got uploaded - https://phabricator.wikimedia.org/T281780 (10AntiCompositeNumber) [19:07:14] 10Traffic, 10SRE-swift-storage, 10Thumbor: Thumbnail of deleted image shown in "File history" after new image with same filename got uploaded - https://phabricator.wikimedia.org/T281780 (10AntiCompositeNumber) File:Benjamin Ola Akande.jpg was deleted after this task was filed, but the [[https://upload.wikime...