[01:27:40] (VarnishHighThreadCount) firing: (8) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [01:32:40] (VarnishHighThreadCount) firing: (8) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [01:37:40] (VarnishHighThreadCount) firing: (9) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [01:42:40] (VarnishHighThreadCount) firing: (16) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [01:57:40] (VarnishHighThreadCount) firing: (12) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [02:02:40] (VarnishHighThreadCount) resolved: (8) Varnish's thread count on cp5017:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [05:53:37] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) The databases are ready to be moved any time. [08:04:11] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A2 from asw-a2-codfw to lsw1-a2-codfw - https://phabricator.wikimedia.org/T355861 (10MoritzMuehlenhoff) I've kicked off a rebalance of ganeti/A now that the maintenance is over. [09:43:27] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [09:44:11] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A2 from asw-a2-codfw to lsw1-a2-codfw - https://phabricator.wikimedia.org/T355861 (10cmooney) 05Open→03Resolved a:03cmooney >>! In T355861#9523826, @MoritzMuehlenhoff wrote: > I've kicked... [10:33:54] hello - I did the pybal restarts before puppet had propagated changes yesterday and need to run them again. Would that be alright? [10:35:00] hnowlan: ok for me [10:35:14] thanks! [13:53:22] 10Traffic, 10Data Products, 10Data-Engineering, 10Observability-Logging, 10Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117 (10gmodena) >>! In T351117#9521093, @Fabfur wrote: > Some updates about the ongoing work: Hey @Fabfur, thanks for this! Blo... [14:52:12] 10Traffic, 10Data Products, 10Data-Engineering, 10Observability-Logging, 10Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117 (10Fabfur) Some updates: * For **backend**, **dt**, **http_status**, **ip**, **response_size** keys, they are now aligned t... [15:20:05] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10ssingh) moss-be* hosts should be @MatthewVernon unless I am mistaken, in which case, please accept my apologies in advance :) [15:21:37] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) >>! In T355544#9525282, @ssingh wrote: > moss-be* hosts should be @MatthewVernon unless I am mistaken, in which case, please accept my... [15:40:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10MatthewVernon) >>! In T355544#9525282, @ssingh wrote: > moss-be* hosts should be @MatthewVernon unless I am mistaken, in which case, please acce... [15:43:25] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [15:48:26] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A7 from asw-a7-codfw to lsw1-a7-codfw - https://phabricator.wikimedia.org/T355867 (10cmooney) >>! In T355867#9498001, @MatthewVernon wrote: > Once complete I'll want to check the backends, but t... [15:50:22] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10cmooney) >>! In T355862#9523604, @Marostegui wrote: > The databases are ready to be moved any time. Great, thanks! [16:09:41] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a24ae7f4-1952-434f-9ee8-3ff0973f1444) set by cmooney@cumin... [16:10:23] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=06c4fbb3-382e-4660-b308-79bf9f5106d5) set by cmooney@cumin... [16:12:04] 10Traffic, 10SRE: PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - https://phabricator.wikimedia.org/T356951 (10LSobanski) [16:28:02] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10ssingh) [16:29:04] vgutierrez: to follow up on yesterday there is some good info on this page as to how everything interacts [16:29:05] https://superuser.com/questions/1787365/debian-network-config-via-systemd-where-do-the-ifup-instances-get-set [16:30:14] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10ssingh) As discussed in [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/998431 | 998431 ]], Traffic will be taking care of `conf2004`, s... [16:31:55] Ultimately this udev rule picks up interfaces that are configured with "allow-hotplug" [16:31:59] cmooney@cumin1002:~$ grep hotplug /lib/udev/rules.d/* [16:31:59] /lib/udev/rules.d/80-ifupdown.rules:# Handle allow-hotplug interfaces [16:31:59] /lib/udev/rules.d/80-ifupdown.rules:SUBSYSTEM=="net", ACTION=="add|remove", RUN+="ifupdown-hotplug" [16:33:20] topranks: oh nice [16:33:27] So it runs /usr/lib/udev/ifupdown-hotplug when the device is created [16:35:08] And that runs systemctl -start $interface [16:36:46] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10cmooney) Work completed! No errors to report all working well. [16:39:18] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) Thanks - I am starting to repool the databases. [17:16:47] 10Traffic, 10Data Products, 10Data-Engineering, 10Observability-Logging, 10Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117 (10xcollazo) >The sequence field now is determined as timestamp + request counter. Even if HAProxy restarts, the timestamp se... [17:27:07] 10Traffic, 10Data Products, 10Data-Engineering, 10Observability-Logging, 10Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117 (10Fabfur) @xcollazo Unfortunately the **timestamp** is referred to the request timestamp. I can check if I can somehow use t... [17:53:40] 10Traffic, 10Data Products, 10Data-Engineering, 10Observability-Logging, 10Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117 (10Fabfur) I've adapted the Benthos configuration to produce an output similar to the current (webrequest) data: ` { "acc... [23:22:40] (VarnishHighThreadCount) firing: (2) Varnish's thread count on cp1102:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:27:40] (VarnishHighThreadCount) firing: (15) Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:32:40] (VarnishHighThreadCount) firing: (15) Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:37:40] (VarnishHighThreadCount) firing: (27) Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:47:41] (VarnishHighThreadCount) firing: (26) Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [23:57:40] (VarnishHighThreadCount) resolved: (13) Varnish's thread count on cp1102:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount