[05:57:03] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Update network SSH keys to ssh-ed25519 - https://phabricator.wikimedia.org/T336769 (10ayounsi) [09:01:33] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: cloudservices2004-dev: reimage into new network setup - https://phabricator.wikimedia.org/T338778 (10aborrero) Seeing many errors like this: ` Jun 19 09:00:07 cloudservices2004-dev pdns_server[1181224]: Received NOTIFY for codfw1dev... [09:32:40] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: cloudservices2004-dev: reimage into new network setup - https://phabricator.wikimedia.org/T338778 (10aborrero) >>! In T338778#8945972, @aborrero wrote: > Seeing many errors like this: > > ` > Jun 19 09:00:07 cloudservices2004-dev pd... [11:29:52] 10Traffic, 10Content-Transform-Team-WIP, 10RESTBase, 10RESTbase Sunsetting, and 6 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365 (10MSantos) [11:48:20] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [11:48:32] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Migrate group0 to Kubernetes - https://phabricator.wikimedia.org/T337490 (10Clement_Goubert) 05Stalled→03In progress [11:59:57] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: cloudservices2004-dev: reimage into new network setup - https://phabricator.wikimedia.org/T338778 (10cmooney) >>! In T338778#8946041, @aborrero wrote: > Fixed by running this in the pdns database; > > ` > update domains set master=... [12:17:18] 10netops, 10Infrastructure-Foundations, 10SRE: Configure QoS marking and policy across network - https://phabricator.wikimedia.org/T339850 (10cmooney) p:05Triage→03Medium [12:34:35] fabfur: o/ ok if I proceed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/930633 ? [12:34:42] cc: btullis: --^ [12:36:07] elukey: Absolutely fine by me. [12:37:04] btullis: one thing that I am wondering atm is about the procedure when the certs will expire, namely I am 99% confident that it will work (puppet generates the new cert, varnishkafka-all is restarted) [12:37:29] but I am wondering if we should keep only ulsfo for the time being to limit the blast radious [12:37:45] ETOOPARANOID? [12:39:27] ah no wow [12:39:28] Not After : Jun 12 06:54:00 2024 GMT [12:39:35] Well, we have a year to wait. [12:39:37] it could be too much [12:39:44] That's just what I was looking at. [12:39:46] I thought it was less, 1y seems long yes [12:40:23] so in theory since we have done ulsfo days before eqsin, if anything explode it should give us some limited blast radious [12:40:31] *explodes [12:40:52] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: cloudservices2004-dev: reimage into new network setup - https://phabricator.wikimedia.org/T338778 (10aborrero) >>! In T338778#8946438, @cmooney wrote: >>>! In T338778#8946041, @aborrero wrote: >> Fixed by running this in the pdns da... [12:40:56] so we wouldn't have all vk instances trying to pull their new certs at once [12:41:30] Yes, maybe just putting something in the calendar before ulsfo expires next year will be enough of a prompt to be ready in case of explosions. [12:41:49] lemme send us a gcal [12:42:23] well no it is surely prone to fail, who knows what we'll do in a year [12:42:37] maybe I can create an ad hoc alarm for this [12:43:03] ok proceeding with eqsin when Traffic will give me the green light [12:43:10] seems safe enough [12:55:46] elukey: I think is safe to proceed [12:56:07] super thanks [12:57:54] 10netops, 10Infrastructure-Foundations, 10SRE: Configure ECMP hashing function on QFX5120 platform - https://phabricator.wikimedia.org/T339852 (10cmooney) p:05Triage→03Medium [14:29:33] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Update network SSH keys to ssh-ed25519 - https://phabricator.wikimedia.org/T336769 (10ayounsi) [14:38:27] 10netops, 10Infrastructure-Foundations, 10SRE: Configure ECMP hashing function on QFX5120 platform - https://phabricator.wikimedia.org/T339852 (10ayounsi) Not tested but looks like the syntax changed slightly to: ` set forwarding-options enhanced-hash-key inet ? Possible completions: + apply-grou... [16:49:27] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Update network SSH keys to ssh-ed25519 - https://phabricator.wikimedia.org/T336769 (10ayounsi) [17:50:37] 10Traffic, 10SRE: Create a cookbook to reboot CDN hosts - https://phabricator.wikimedia.org/T338813 (10BCornwall) 05In progress→03Resolved [17:58:16] 10Traffic, 10DNS: add wikimedia.social to WMF DNS (was: Update DNS records for mastodon.wikimedia.org) - https://phabricator.wikimedia.org/T337586 (10BCornwall) 05Open→03In progress a:03BCornwall [17:58:57] 10Traffic, 10DNS: add wikimedia.social to WMF DNS (was: Update DNS records for mastodon.wikimedia.org) - https://phabricator.wikimedia.org/T337586 (10BCornwall) [18:25:54] 10Traffic, 10DNS, 10Patch-For-Review: add wikimedia.social to WMF DNS (was: Update DNS records for mastodon.wikimedia.org) - https://phabricator.wikimedia.org/T337586 (10BCornwall) Ignore the gerritbot comment, it's been updated to use wikimedia.social. @Dzahn I'd love your re-review not only for the technic... [20:20:27] 10Traffic, 10DNS, 10Patch-For-Review: add wikimedia.social to WMF DNS (was: Update DNS records for mastodon.wikimedia.org) - https://phabricator.wikimedia.org/T337586 (10BCornwall) 05In progress→03Resolved ` $ drill -Q @ns{0..2}.wikimedia.org wikimedia.social 178.33.220.142 176.31.213.231 ` All set!