[07:06:19] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Zuul Zookeeper does not generate logs - https://phabricator.wikimedia.org/T435503#12326139 (10hashar) Confirmed, thx for the deploy!  Note from https://gerrit.wikimedia.org/r/c/operations/puppet/+/1327569/comments/70c86841_bb...
[07:25:59] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Addition of zuul1004 broke zuul-eqiad zookeeper - https://phabricator.wikimedia.org/T435186#12326168 (10hashar) Now that we have Zookeeper logs on the zuul* hosts (T435503), I can see the connection failures: ` name=zuul1001.eqia...
[08:12:11] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Addition of zuul1004 broke zuul-eqiad zookeeper - https://phabricator.wikimedia.org/T435186#12326351 (10hashar) Found it, the client port is //wrong//  https://gerrit.wikimedia.org/r/c/operations/puppet/+/1236730 //zookeeper/zuul...
[08:13:18] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Addition of zuul1004 broke zuul-eqiad zookeeper - https://phabricator.wikimedia.org/T435186#12326358 (10hashar)  Looking at the firewall rules on `zuul1001` using `sudo iptables -L -v|grep dpt:2.8`:  We get rules for all three ho...
[16:20:42] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Zuul Zookeeper does not generate logs - https://phabricator.wikimedia.org/T435503#12329557 (10Dzahn) Thanks for confirming!  :)  I would prefer if that is handled separately as this task was just about zuul machines and indee...
[16:21:23] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Zuul Zookeeper does not generate logs - https://phabricator.wikimedia.org/T435503#12329560 (10Dzahn) >>! In T435503#12326139, @hashar wrote: > For the Zuul Zookeeper instances, we now have logs, I can see connection errors an...
[21:18:15] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Addition of zuul1004 broke zuul-eqiad zookeeper - https://phabricator.wikimedia.org/T435186#12331033 (10Dzahn) 172.17.0.0/16 is allowed from this puppet code:   `      # allow traffic between zuul services in docker and zookeeper...
[21:19:54] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Addition of zuul1004 broke zuul-eqiad zookeeper - https://phabricator.wikimedia.org/T435186#12331035 (10Dzahn) Before anything else - I suggest we talk in a group setting about how exactly the physical machines are replacing (or...