[00:25:46] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5026.eqsin.wmnet with OS buster [00:48:52] 10Domains, 10Cloud-Services, 10Education-Program-Dashboard, 10SRE, 10Traffic-Icebox: Create short link for outreachdashboard.wmflabs.org - https://phabricator.wikimedia.org/T146332 (10Dzahn) [00:51:17] 10Domains, 10Cloud-Services, 10Education-Program-Dashboard, 10SRE, 10Traffic-Icebox: Create short link for outreachdashboard.wmflabs.org - https://phabricator.wikimedia.org/T146332 (10Dzahn) status here as of today is: https://dash.wmflabs.org/ exists but shows an error because no proxy is configured h... [00:52:48] 10Domains, 10Cloud-Services, 10Education-Program-Dashboard, 10SRE, 10Traffic-Icebox: Create short link for outreachdashboard.wmflabs.org - https://phabricator.wikimedia.org/T146332 (10Dzahn) I added Cloud-Service because it seems to me this needs an admin from https://openstack-browser.toolforge.org/proj... [01:25:05] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5026.eqsin.wmnet with OS buster completed: - cp5026 (**PASS**) -... [01:27:11] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5027.eqsin.wmnet with OS buster [02:21:09] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5027.eqsin.wmnet with OS buster executed with errors: - cp5027 (**... [02:21:37] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5027.eqsin.wmnet with OS buster [02:22:01] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5027.eqsin.wmnet with OS buster executed with errors: - cp5027 (**... [02:22:26] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5027.eqsin.wmnet with OS buster [02:33:30] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5027.eqsin.wmnet with OS buster executed with errors: - cp5027 (**... [10:19:59] vgutierrez: how does https://gerrit.wikimedia.org/r/c/operations/puppet/+/861825 look to you? I'll merge if all good [10:23:22] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10cmooney) @dcaro also advised me this can be set for the monitor traffic also, with 'mon_use_min_delay_socket': https://github.com/c... [10:44:04] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Joe) For the record, we decided to start with option 3 and we're starting with rollout phase 1, specifically we'll move test2.wikipedia.org to kubernetes first. [11:37:22] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10BTullis) @cmooney - This looks very useful. We can certainly look at using `osd_heartbeat_use_min_delay_socket=true` from the outset... [11:37:49] godog: +1ed a awhile ago :) [12:37:29] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10MoritzMuehlenhoff) ganeti5004 has been added to the eqsin Ganeti cluster. [14:44:16] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ssingh) [15:38:21] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: `dns5001.wikimedia.org` - dns5001.wikimedia.... [15:39:21] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ssingh) [15:45:42] 10Traffic, 10serviceops, 10Platform Team Initiatives (API Gateway): Handle edge cache invalidation for the api gateway - https://phabricator.wikimedia.org/T324200 (10Vgutierrez) Re-tagging the task, I'm assuming it got into traffic-icebox by mistake :) [17:15:05] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host dns5004.wikimedia.org with OS buster [17:47:43] 10Traffic, 10SRE, 10serviceops, 10Platform Team Initiatives (API Gateway): Handle edge cache invalidation for the api gateway - https://phabricator.wikimedia.org/T324200 (10daniel) Note that we only need active purging if/when we emit cache control headers that tell the edge case to cache long-term. One k... [18:01:45] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host lvs5004.eqsin.wmnet with OS buster [19:27:36] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host lvs5004.eqsin.wmnet with OS buster executed with errors: - lvs5004 (... [19:39:50] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host dns5004.wikimedia.org with OS buster executed with errors: - dns5004... [19:40:24] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host dns5004.wikimedia.org with OS buster