[00:50:02] <Krinkle>	 Wordpress perf team taking shape: https://make.wordpress.org/core/2021/10/28/wordpress-performance-team-kick-off/
[00:52:08] <Krinkle>	 Spreadsheet includes focus on improvements to use of memc and redis, SQL optimisation and abstraction,  more native localisation logic, CDN integrations
[07:39:49] <XioNoX>	 I'm merging a patch to add drmrs to monitoring, please ignore if you see a related alert
[09:38:25] <gehel>	 I'm planning to do a presentation of how we use Elasticsearch. Mostly for Emmanuel who just joined our team. But if anyone else wants to join, that would be nice! Ideally in the European morning. Do we have any interested new hires?
[09:38:51] <btullis>	 Yes please. 
[09:43:02] <gehel>	 btullis: I'll add you to it!
[09:43:31] <btullis>	 gehel: Many thanks. 
[09:44:17] <gehel>	 Monday Nov 8, 10:30am CET
[09:44:24] <gehel>	 feel free to forward to anyone
[10:22:50] <_joe_>	 gehel: yes
[10:31:58] <gehel>	 _joe_: yes as in you'll forward it ? Or as in you want to be there yourself ?
[10:32:27] <_joe_>	 gehel: I guess our most recent hires could benefit from it
[10:32:36] * _joe_ looks at jelto and arnoldokoth
[10:32:37] <_joe_>	 :)
[10:37:09] * jelto would be happy to get a forward of the invite
[10:37:56] <jelto>	 I thinks it's outside of arnoldokoth working window, so it would be good to record the session if thats fine for you all
[10:41:36] <dcaro>	 gehel: I'm not a new hire, but I'm interested too :)
[11:39:34] <jayme>	 gehel: I'd like to join as well o/
[11:42:17] <gehel>	 invite sent to all the above
[11:43:19] <gehel>	 there is a previous version of that talk recorded and available in the SRE Onboarding chats folder: https://drive.google.com/file/d/18z5i6OgRLsWsn95dQGJxbWq6yVBzG_nK/view?usp=sharing
[12:22:50] <Emperor>	 gehel: (what's that in UTC, sorry?) I'd be interested, assuming it doesn't clash with my team meeting
[12:24:59] <gehel>	 Emperor: that's 10:30 CET, which should be 9:30 UTC
[12:26:34] <gehel>	 Emperor: I've sent you the invite, feel free to ignore it
[12:27:00] <Emperor>	 Thanks - that should fit in pretty well :)
[12:27:53] <gehel>	 Cool!
[13:50:33] <ottomata>	 godog: o/ trying out pontoon for the first time
[13:50:40] <ottomata>	 just tried to run bootstrap.sh on a new cloud vps host
[13:50:45] <ottomata>	 got an error 
[13:50:57] <ottomata>	 Error: Could not find resource 'Package[puppet]' in parameter 'require' (file: /tmp/bootstrap/git/puppet/modules/profile/manifests/puppetmaster/pontoon.pp, line: 117) on node pontoon-1.analytics.eqiad1.wikimedia.cloud
[13:50:57] <ottomata>	 install: cannot create regular file '/etc/puppet/hieradata/auto.yaml': No such file or directory
[13:51:40] <ottomata>	 got a few Warnings above that too
[13:51:56] <Spookreeeno>	 i imagine the 2nd one is because of the first
[13:52:08] <Spookreeeno>	 and probably shouldn't run if no puppet
[13:55:22] <ottomata>	 commented out that require on line 117 in /tmp/bootstrap, now getting:
[13:55:22] <ottomata>	 Error: Could not find resource 'Exec[compile puppet.conf]' in parameter 'notify' (file: /tmp/bootstrap/git/puppet/modules/base/manifests/puppet/config.pp, line: 22) on node pontoon-1.analytics.eqiad1.wikimedia.cloud
[13:55:56] <ottomata>	 i guess we need to include base::puppet?>
[13:57:20] <godog>	 ottomata: hi, thank you for trying out pontoon! it is possible some bits are missing, I haven't tested a bootstrap yet after the recent base refactorings
[13:57:26] <ottomata>	 right
[13:57:48] <ottomata>	 godog:  manually edit pontoon.pp including base::puppet... it seems to be running now
[13:58:37] <ottomata>	 dunno if the default params are right
[13:58:39] <godog>	 ottomata: nice! please report progress/problems, I'll be on vacation starting next week but I'll take a look once I'm back
[13:58:40] <ottomata>	 will submit patch if it works
[13:59:42] <godog>	 ack, yeah for tasks feel free to tag "user-fgiunchedi" for now
[13:59:59] <ottomata>	 i think its working, just submitting patch
[14:00:49] <ottomata>	 godog: https://gerrit.wikimedia.org/r/c/operations/puppet/+/735650/
[14:01:39] <godog>	 neat, thanks!
[14:01:44] <ottomata>	 getting other errors now...
[14:01:53] <ottomata>	 but i think it worked...
[14:01:54] <ottomata>	 ?
[14:02:07] <ottomata>	 E: Failed to fetch http://deb.debian.org/debian/pool/main/d/docopt/python3-docopt_0.6.2-2_all.deb  Cannot initiate the connection to webproxy.eqiad.wmnet:8080 (2620:0:861:1:208:80:154:32). - connect (101: Network is unreachable) Could not connect to webproxy.eqiad.wmnet:8080 (208.80.154.32), connection timed out
[14:03:07] <godog>	 yeah it might be that some variables moved places, looks like the proxy is getting set but that's obviously not what needs to happen in cloud vps
[14:03:12] <ottomata>	 via Notice: /Stage[main]/Cergen/Apt::Package_from_component[cergen]/Exec[exec_apt_cergen]: Triggered 'refresh' from 1 event
[14:03:16] <ottomata>	 right...
[14:04:20] <godog>	 ottomata: not the best timing now unfortunately, I'm aiming for an "zero touch" bootstrap but that clearly needs work post-refactoring
[14:05:18] <godog>	 bbiabi
[14:05:25] <ottomata>	 its class apt
[14:05:27] <ottomata>	 has $use_proxy => true
[14:05:32] <ottomata>	 and hardcoded proxies to use if so
[14:06:15] <ottomata>	 dunno why all the sudden not working, been like that for a while
[14:06:29] <ottomata>	 patching..
[14:08:36] <ottomata>	 godog: if i add hiera to pontoon.yaml
[14:08:45] <ottomata>	 willl that take precedence?
[14:08:49] <ottomata>	 oh wait i htink you have docs...
[14:09:00] <ottomata>	 hieradata/pontoon.yaml
[14:09:00] <ottomata>	 Common to all Pontoon stacks, changes to this file shouldn't be needed.
[14:09:01] <ottomata>	 :)
[14:10:19] <ottomata>	 godog:  Q: is it best to leave the pontoon host's puppetmaster to be the default cloud puppetmaster?
[14:10:28] <ottomata>	 and only use the pontoon host as puppet master for other hosts?
[14:10:32] <ottomata>	 i guess so, right?
[14:11:00] <ottomata>	 or, does bootstrap automaically make it a puppet client of itself?
[14:17:46] <godog>	 ottomata: yeah at the end of bootstrap the server is a client of itself
[14:17:55] <ottomata>	 ok
[14:18:04] <godog>	 essentially ready to go to enroll new hosts
[14:18:52] <ottomata>	 godog:  can i run bootstrap multipe times?  i think maybe it failed some automated cert signing because apt failed to install cergen because http proxy was set wrong for apt?
[14:19:19] <ottomata>	 certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet CA: puppet]
[14:20:15] <godog>	 ottomata: not sure if bootstrap is idempotent but you could try yeah
[14:20:58] <godog>	 but no cergen isn't involved in bootstrap, maybe the bootstrap of the master itself failed
[14:21:17] <ottomata>	 yeah ok i think that's it then
[14:21:42] <ottomata>	 hm yeah no dunno
[14:21:50] <ottomata>	 bootstrapped seemd to work this time
[14:22:01] <ottomata>	 but puppet agent can't run 
[14:23:18] <godog>	 yeah I'm sorry I can't be more helpful atm, I'll take a closer look when I'm back from vacation
[14:24:47] <ottomata>	 s'ok
[14:31:04] <ottomata>	 godog: FYI, I was able to enroll a new host, but had to manually sign the certificate on the pontoon puppetmaster.  after that, puppet failling because the host is not known to puppet (somehow the rolemap isn't working?)
[14:31:46] <ottomata>	 ok, gonna give up for now. i'm really excited to be able to use this, lemme know  after your vaca if i can help in any way!
[14:32:23] <ottomata>	 hmm
[14:32:27] <ottomata>	 it is using the wrong rolemap?
[14:32:29] <ottomata>	  ERROR:root:Host an-db-1.analytics.eqiad1.wikimedia.cloud not found in /var/lib/git/operations/puppet/modules/pontoon/files/bootstrap/rolemap.yaml
[14:34:21] <godog>	 ottomata: mmhh did bootstrap write the instructions on next steps in /etc/README.pontoon on the server?
[14:34:36] <godog>	 there a few additional steps to complete after running bootstrap.sh
[14:35:30] <ottomata>	 i  think I ran them based on the output of bootstreap
[14:35:31] <ottomata>	 checking readme
[14:35:59] <ottomata>	 hmmmm
[14:36:03] <ottomata>	  /etc/pontoon-stack has wong content
[14:36:06] <ottomata>	 i did run that though...mabye it failed?
[14:36:31] <ottomata>	 ok now it has right content
[14:36:34] <ottomata>	 guess i missed it
[14:37:04] <ottomata>	 looks like that worked
[14:37:12] <ottomata>	 at least for the client node!
[14:37:15] <ottomata>	 continuing then :)
[14:37:16] <ottomata>	 ty
[14:37:39] <godog>	 ottomata: sure np! thanks for the feedback, I'll be around another 1.5h or so
[20:09:17] <ottomata>	 i guess a new node is not added to puppet db until it has a real role that is not insetup?
[20:09:19] <ottomata>	 jbond: do you know^ ?
[20:09:27] <ottomata>	 i'd like to run pcc on e.g. an-db1001
[20:09:32] <ottomata>	 but it is currently insetup
[20:11:48] <jbond>	 ottomata: if i understand correctly yu shuld be able to send a CR updating site.pp to move an-db1001 to the role you want.  then run pcc and you will see what will get applied.  allthough it would likley be quite big
[20:12:00] <ottomata>	 yeah, i've got a CR doing that
[20:12:09] <ottomata>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/735688
[20:12:14] <ottomata>	 but PCC is giving me
[20:12:15] <jbond>	 oh you may need to run the updat facts script
[20:12:27] <ottomata>	 oh?
[20:12:28] <ottomata>	 OH
[20:12:29] <ottomata>	 RIGHT
[20:13:00] <ottomata>	 wait no, where is that?
[20:13:31] <jbond>	 https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler#Updating_nodes
[20:14:03] <ottomata>	 gr8 trying
[20:14:26] <jbond>	 cool going back of line but ping if you still ave issues
[20:14:39] <ottomata>	 jbond:  i don't have access to compiler project
[20:14:45] <jbond>	 one sec
[20:14:49] <ottomata>	 sorry (didn't realize you were offline, what's your tz?)
[20:16:42] <jbond>	 should be added now, im one CET but its not a problem
[20:18:02] <ottomata>	 jbond:  getting Unable to find fact file for: an-db1001.eqiad.wmnet
[20:18:18] <ottomata>	 or, yo umeant added to project
[20:18:25] <ottomata>	 ah i think i can run now...maybe?
[20:18:38] <jbond>	 yrd added to the project you shuld be able to run the script now
[20:18:54] <jbond>	 it takes a few minutes so kick it of and make a cup of tea ;)
[20:19:09] <ottomata>	 ok, now i'm having some i think local ruby issues...
[20:19:16] <ottomata>	 but i think i can ssh in so will debug here
[20:19:19] <ottomata>	 thanks jbond !
[20:19:40] <jbond>	 on sec let me kick it of for you
[20:19:45] <ottomata>	 ty
[20:25:25] <ottomata>	 any luck jbond ?
[20:25:41] <jbond>	 ottomata: still running :) 
[20:30:02] <jbond>	 ottomata: try now
[20:30:27] <ottomata>	 trying
[20:31:11] <ottomata>	 YES ty jbond 
[20:31:14] <ottomata>	 https://puppet-compiler.wmflabs.org/compiler1001/32017/
[20:32:08] <jbond>	 cool :)