[07:50:44] 10Traffic, 10API Platform, 10SRE: Block non-browser requests that use generic user agent (UA) headers - https://phabricator.wikimedia.org/T319423 (10Joe) [09:32:15] brett: I've spotted the issue with that CR; I've reuploaded it as https://gerrit.wikimedia.org/r/c/operations/puppet/+/855494 and it's ready for your review [09:45:40] 10Traffic, 10Data Pipelines, 10Data-Engineering-Planning, 10Foundational Technology Requests, and 2 others: Add a webrequest sampled topic and ingest into druid/turnilo - https://phabricator.wikimedia.org/T314981 (10elukey) Assumptions/notes for the first prototype: * Benthos reads webrequest_{text,upload... [10:26:56] 10Traffic, 10Data Pipelines, 10Data-Engineering-Planning, 10Foundational Technology Requests, and 2 others: Add a webrequest sampled topic and ingest into druid/turnilo - https://phabricator.wikimedia.org/T314981 (10JAllemandou) happy to talk about sampling when you wish @elukey :) I also think you should... [10:34:07] 10Traffic, 10Data Pipelines, 10Data-Engineering-Planning, 10Foundational Technology Requests, and 2 others: Add a webrequest sampled topic and ingest into druid/turnilo - https://phabricator.wikimedia.org/T314981 (10elukey) >>! In T314981#8385855, @JAllemandou wrote: > happy to talk about sampling when you... [10:49:59] 10Traffic, 10Data Pipelines, 10Data-Engineering-Planning, 10Foundational Technology Requests, and 2 others: Add a webrequest sampled topic and ingest into druid/turnilo - https://phabricator.wikimedia.org/T314981 (10JAllemandou) Something I forgot: It'd be great to have the new stream setup as an "event" (... [10:53:25] 10Traffic, 10Data Pipelines, 10Data-Engineering-Planning, 10Foundational Technology Requests, and 2 others: Add a webrequest sampled topic and ingest into druid/turnilo - https://phabricator.wikimedia.org/T314981 (10JAllemandou) >> I also think you should be pointed to T319344, as it requests a new feature... [11:57:53] 10Traffic, 10Analytics-Radar, 10SRE, 10Patch-For-Review: Consider adding X-Analytics subfield for 'has a session cookie' - https://phabricator.wikimedia.org/T319324 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez CR merged and https://wikitech.wikimedia.org/wiki/X-Analytics#Keys updated, thanks for c... [12:24:51] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet - https://phabricator.wikimedia.org/T296411 (10aborrero) [12:26:08] 10netops, 10Infrastructure-Foundations, 10cloud-services-team (Kanban): connect 2nd cloudcontrol200x-dev NIC to vlan 2105 - https://phabricator.wikimedia.org/T297588 (10aborrero) 05Stalled→03Declined It seems we'll be working on {T297596} instead. [12:26:32] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet - https://phabricator.wikimedia.org/T296411 (10aborrero) [14:14:00] vgutierrez: Thanks! I suspected that might be the issue. I'm a little confused as to why pcc didn't catch it :/ [14:16:01] I disagree that's a soluton [14:16:14] it will generate duplicated resources as it is [14:16:32] a define can be called multiple times with different $titles [14:16:45] but both file { $exec: and systemd::timer::job { 'prometheus-trafficserver-config': [14:17:14] have static names that don't depend on the title and will generate duplicated resources that will make puppet fail, unless I'm missing something [14:17:25] vgutierrez, brett ^^^ [14:17:52] I know I know [14:18:45] that will be handled at some point by https://gerrit.wikimedia.org/r/c/operations/puppet/+/851669 [14:19:09] brett missed the fact that we support several ATS instances per hjost [14:19:10] *host [14:20:40] is prometheus::node_trafficserver_config called twice on some host? [14:20:47] because if so I think your patch will fail [14:23:57] not right now [14:24:05] cause we don't have ats-tls in place anymore [14:24:25] that's why it got merged as a class and stuff didn't melt :) [14:27:08] So we support multiple ATS instances but we don't use them, is that right? [14:28:23] right now that's true [14:28:36] and that's why your original CR didn't break puppet on the cp cluster [14:29:18] but that puppetization needs to be improved as mentioned on https://phabricator.wikimedia.org/T292815#8352835 [14:39:21] vgutierrez: But the original CR *did* break it, didn't it? Wasn't that why it was reverted in the first place? [14:39:51] though it was just for syntax reasons rather than multiple ATS instances [14:41:40] reverting it would leave some artifacts on the servers unmanaged.. so I chose the refactor approach [15:16:39] 10Traffic, 10Data Pipelines, 10Data-Engineering-Planning, 10Foundational Technology Requests, and 2 others: Add a webrequest sampled topic and ingest into druid/turnilo - https://phabricator.wikimedia.org/T314981 (10Ottomata) > overridden by the DE batch jobs QQ: do the existent DE batch jobs already produ... [15:33:20] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin2002 for host cloudgw2002-dev.codfw.wmnet with... [16:01:29] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) [16:01:43] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) [16:04:02] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin2002 for host cloudgw2002-dev.codfw.wmnet with OS... [16:25:00] 10Traffic, 10Data Pipelines, 10Data-Engineering-Planning, 10Foundational Technology Requests, and 2 others: Add a webrequest sampled topic and ingest into druid/turnilo - https://phabricator.wikimedia.org/T314981 (10elukey) >>! In T314981#8386970, @Ottomata wrote: >> overridden by the DE batch jobs > QQ: d... [16:27:55] 10Traffic, 10Data Pipelines, 10Data-Engineering-Planning, 10Foundational Technology Requests, and 2 others: Add a webrequest sampled topic and ingest into druid/turnilo - https://phabricator.wikimedia.org/T314981 (10Ottomata) > The idea is to proceed for a first iteration (namely a new kafka topic + druid... [23:07:16] (VarnishChildRestarted) firing: varnish-upload restarted on cp4047 - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000330/varnish-machine-stats?orgId=1&viewPanel=66&var-server=cp4047&datasource=ulsfo%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DVarnishChildRestarted