[07:46:27] <XioNoX>	 can I get a "just in case" review of https://gerrit.wikimedia.org/r/c/operations/dns/+/830085 for the upcoming maintenance?
[07:48:17] <godog>	 {{done}}
[07:49:48] <XioNoX>	 thx1
[07:53:03] <XioNoX>	 FYI, ulsfo routers upgrade starting in ~10min
[09:09:31] <jnuche>	 hi there! we have a constant stream of DB errors for user wikiuser202206 since 6am UTC this morning. I don't really know how critical it is but it didn't look too healthy so I wanted to mention it just in case: https://logstash.wikimedia.org/goto/06e4660ffae3824e45c8e0966743032f
[09:10:04] <marostegui>	 jnuche: thanks, looking
[09:10:46] <marostegui>	 Amir1: That's probably grants issue on the new x1 eqiad master when comming from codfw
[09:11:10] <marostegui>	 Fixing it now
[09:11:22] <Amir1>	 ah, interesting 
[09:11:42] <Amir1>	 thanks. We should have it in puppet
[09:12:30] <marostegui>	 Fixed, let's see if the errors go away
[09:12:56] <marostegui>	 db1103 (old master) was ok
[09:35:53] <XioNoX>	 Router upgrade is hitting a bug (the embedded linecard is not booting up), so ulsfo will stay depooled for longer than expected
[09:37:12] <marostegui>	 The error has stopped
[09:37:20] <marostegui>	 jnuche: ^ thanks for letting us know!
[09:37:46] <addshore>	 XioNoX: following on from yesterday, what uplink does wdqs1004 have?
[09:37:47] <jnuche>	 marostegui: np!
[09:38:45] <XioNoX>	 addshore: 1G as well
[09:39:08] <addshore>	 XioNoX: thanks, next time I might just transfer it straight from wdqs1004 -> google cloud rather than via a stat machine
[09:39:42] <dcausse>	 addshore: wdqs1004 is a production machine so no :)
[09:39:52] <addshore>	 bah *re checks his notes*
[09:40:06] <addshore>	 im confusing stat1004 and wdqs1009!
[09:40:17] <addshore>	 XioNoX: does wdqs1009 also have 1G?
[09:41:19] <addshore>	 dcausse: looking at the timings i can get it out of the cluster about as fast as moving it within the cluster. so next time I try this I'll get get puppet disabled, turn the service off, copy it myself, and when its done get puppet turned back on
[09:41:40] <addshore>	 I don't really see myself trying to do this more than around once a month right now
[09:44:13] <dcausse>	 addshore: ok then we might want to have a cookbook to avoid manual steps
[09:44:50] <addshore>	 sounds good to me, the steps should be relatively easy
[09:45:13] <addshore>	 will chat about it a little with you in the coming months / write down the steps etc from my side
[09:45:24] <addshore>	 but next job is to check if this file works for folks on the outside / is sueful
[09:47:04] <dcausse>	 sure
[09:47:52] <XioNoX>	 addshore: yep 1G too
[09:48:06] <XioNoX>	 if you have netbox access you can look it up: https://netbox.wikimedia.org/dcim/devices/986/interfaces/
[09:48:58] <XioNoX>	 or on the host itself:
[09:49:00] <XioNoX>	 wdqs1009:~$ sudo ethtool eno1 | grep Speed
[09:49:00] <XioNoX>	 	Speed: 1000Mb/s
[09:49:25] <XioNoX>	 if you have sudo there
[09:50:06] <addshore>	 *checks*
[09:50:14] <addshore>	 oooh yes, I see netbox, lovely!
[11:38:45] <XioNoX>	 ulsfo routers upgrade is done, there are a few wrinkles to iron out
[11:38:55] <XioNoX>	 but overall the site is ready for traffic again
[11:39:01] <XioNoX>	 I'll let it sit a bit then repool
[12:19:19] <moritzm>	 FYI, going to disable puppet in codfw and the edges for 10-15 starting in 5m
[12:30:11] <claime>	 ack
[13:06:43] <ottomata>	 Hiya, we are trying to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/821695 on an-test-client1001.eqiad.wmnet, and getting errors.  btullis merged this 4 hours ago.  the error says it is from file: /etc/puppet/modules/profile/manifests/hadoop/spark3.pp, line: 148.  On puppetmaster1001 i can see this file is indeed incorrect and not the line from the merged commit.  AFAICT puppet-merge has been run.  
[13:06:43] <ottomata>	 /var/lib/git/operations/puppet is up to date.  
[13:09:02] <ottomata>	 so i guess my q is: why is /etc/puppet showing an old file?
[13:33:20] <btullis>	 Is it this line? https://gerrit.wikimedia.org/r/c/operations/puppet/+/821695/7/modules/profile/manifests/hadoop/spark3.pp#152 Should be `content` instead of `source`?
[13:48:41] <ottomata>	 ....?  btullis  i fetched production and i see it as content => ????
[13:51:30] <btullis>	 Oh. What about here? Is this out of date? https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/hadoop/spark3.pp#L152
[13:52:24] <ottomata>	 i guess it is.... must be my local copy is somehow wrong...
[13:53:46] <akosiaris>	 ottomata: same md5 for puppetmaster1001:/etc/puppet/modules/profile/manifests/hadoop/spark3.pp and local modules/profile/manifests/hadoop/spark3.pp btw
[13:53:58] <ottomata>	 yeah......
[13:54:06] <ottomata>	 okay, then maybe my fetch was wrong...
[13:54:08] <ottomata>	 okay
[13:55:44] <ottomata>	 indeed, new fetch and rebase show sit wrong...okay
[13:56:06] <btullis>	 New patch here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/830170
[13:57:07] <ottomata>	 you beat me!
[13:57:56] <ottomata>	 merging btullis 
[13:58:20] <btullis>	 Great, thanks.