[10:08:59] <dcausse>	 the default cluster might be here since we switched to the extension registry and the way it merges config entries with arrays
[10:15:19] <btullis>	 dcausse: Are you happy for me to start the upgrade of an-airflow1005 to bullseye now?
[10:15:32] <dcausse>	 btullis: yes please go ahead :)
[10:15:41] <btullis>	 Ack, thanks.
[11:03:25] <btullis>	 dcausse: The upgrade of an-airflow1005 has completed and all looks OK. There will be one more brief reboot in a few minutes, after the host is also upgraded to puppet 7.
[11:03:43] <dcausse>	 btullis: thanks!
[11:04:00] <btullis>	 a pleasure.
[11:25:37] <dcausse>	 lunch
[12:56:40] <gehel>	 Errand, back soonish 
[13:50:04] <inflatador>	 <o/
[13:53:17] <dcausse>	 o/
[14:49:42] <dcausse>	 quick errand
[15:22:41] <inflatador>	 Heads up on cloudelastic: I'll be merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/993764 shortly. No impact is expected, but there's a small chance that CE would be inaccessible for as long as it takes me to revert the patch (~30m)
[15:49:38] <inflatador>	 interesting...we're getting an SSL cert alert for the canary elastic host...probably a problem with alt names on the cert, but shows up as cert expiration
[16:00:01] <ebernhardson>	 \o
[16:02:03] <inflatador>	 .o/
[16:03:58] <dcausse>	 o/
[16:57:14] <inflatador>	 workout/lunch/errands, back in ~2h
[18:33:15] <dcausse>	 now I get why I see CoolBot/0.0 UA in wdqs query logs (https://meta.wikimedia.org/wiki/User-Agent_policy)
[18:35:14] <ebernhardson>	 oh, lol
[18:35:42] <dcausse>	 it's a literal copy/paste of this ua
[18:58:35] <dcausse>	 dinner
[19:11:28] <gehel>	 That's almost funny ;)
[19:22:44] <inflatador>	 back
[19:34:33] <moritzm>	 ryankemper, inflatador: ok to move wdqs/public to Puppet 7? we have 1018-1021 on Puppet 7 already, so should be without issues
[19:40:48] <inflatador>	 moritzm ACK, feel free to move anytime
[19:43:21] <moritzm>	 k, I'm going ahead now then
[20:01:53] <moritzm>	 wdqs/public is now on Puppet 7
[20:02:02] <gehel>	 moritzm: thanks!
[20:02:20] <inflatador>	 {◕ ◡ ◕}
[20:19:46] <inflatador>	 Our cloudelastic private IP canary joined the cluster! everything looking good so far
[20:51:59] <inflatador>	 restarting cloudelastic cluster again so all hosts have the same running config
[20:53:35] <inflatador>	 aaand, we have a failure on the first host, cloudelastic1003. So the current config is not going to work
[20:55:39] <inflatador>	 oh fun, it's bombing out because there aren't any "site" scoped interfaces on the public IP'd hosts
[21:00:35] <inflatador>	 ryankemper ebernhardson patch up to fix the interface issue: https://gerrit.wikimedia.org/r/c/operations/puppet/+/994321 . using config as described at https://www.elastic.co/guide/en/elasticsearch/reference/7.17/modules-network.html#network-interface-values
[21:34:40] <inflatador>	 OK, one more time on the clusters
[21:48:40] * inflatador means I'm restarting the cloudelastic cluster again
[21:51:22] <ebernhardson>	 fun :)
[21:58:59] <inflatador>	 in better news, the pybal stuff should supposedly work seamlessly with private IP'd hosts, so we won't have to migrate all the load balancer stuff
[21:59:08] <inflatador>	 change DNS, etc
[22:01:13] <ebernhardson>	 thats great news
[22:07:49] <inflatador>	 yeah, my only (minor) concern is that our puppet code is automatically requesting the hostname of the server as an alt name for the letsencrypt certs, since the private IP'd hosts will be in .wmnet domain
[22:19:14] * ebernhardson struggles documenting something painfully obvious...why you must use https with the NetworkAuth extension
[22:44:45] <inflatador>	 'nother small CR around the CE migration https://gerrit.wikimedia.org/r/c/operations/puppet/+/994338
[22:57:48] <ryankemper>	 inflatador: +1'd