[08:53:33] arnaudb: okay to merge your homedir change? [08:53:40] ha ha [08:53:45] :D [08:53:48] I was asking the same on operations- [08:53:59] arnaudb: see if the diff matches and say yes or no [08:54:11] arnaudb: diff matches your own patch? [08:54:17] you want to deploy and it is ready? [08:54:34] and tell jayme [08:56:24] arnaudb/jynus I did not merge now, but my change is good to merge [08:56:41] looks good to me jayme, will merge then :) [08:57:42] this is good, as this will likely happen to you, but always check your patch is the same as the one expected, and there was no weird man in the middle / impresionation [08:57:54] as you just did! [08:58:31] for the new ones if you want a glimpse of the past look at T224033 :D [08:58:31] T224033: Fix operations/puppet.git "rebase hell" - https://phabricator.wikimedia.org/T224033 [09:01:58] the merge should be rolling out jayme ! [09:02:27] arnaudb: cool, thanks! [10:04:27] brouberol: ok to merge your labs/private changes? [10:04:55] yes! [10:05:46] done [10:05:48] FYI labs-private changes, although do not affect production, are propagated to the WMCS environment and do to so need a puppet-merge run so that they use the last "officially" merged one and not the HEAD in gerrit [10:06:16] similar to what puppet-merge does for changes in the puppet repo [10:07:49] was there a hickup when service en.wikipedia.org? for a minute I got timeout when trying to go to https://en.wikipedia.org/wiki/Main_Page, but not https://en.wikipedia.org [10:08:12] s/when/whith [10:10:02] dcaro: I don't see anything in the last few minutes that jumps at the eye or alerts [10:10:28] might have just been my network/isp, thing got weird for a bit [10:15:56] duckduck go does not work, but google works, drive.google does not, but spotify does... I think it's on my side xd [10:16:33] lol [13:47:39] on-call folks: rolling out nsa->ns2 updates across alerting and dns hosts [13:48:07] if it pages, please let me know and I will take care of it, thanks. nothing to worry as such, the IP has already been changed, just doing renaming here [13:51:49] sukhe: ack, thx for the headsup [14:26:58] godog: we have two patches in puppet, you can merge mine [14:27:10] dhinus: cheers! was about to ask [14:27:12] {{done}} [15:56:25] apergos: I'm seeing a puppet alert on clouddumps100[12] -- does the puppet run itself generate some/all of the dumps? [15:56:37] the alert is because puppet almost always does something on those hosts. Maybe that's expected. [15:56:58] clouddumps hosts belong to wmcs [15:57:19] these are the public web server and the host that serves nfs to wmcs instances [15:57:21] I know but usually you know more about them than me :) [15:57:29] uh [15:57:33] ok so [15:58:48] for pings like this, since I am transitioning off of dumps (at last!), I am supposed to redirect people to Virginia Poundstone and Will Doran (both of Data Engineering, that team is taking them over) and also to point people to https://www.mediawiki.org/wiki/Data_Platform_Engineering/Data_Products/Intake_Process for some general information [15:59:01] having said that, what are these alerts? [15:59:12] theyu won't have to do with dumps generation, that happens on other hosts [15:59:35] "Puppet performing a change on every puppet run" [15:59:51] I'm also pingable for dumps related SRE stuff (although Will and Virginia have asked to be notified, as apergos mentioned). [16:00:00] what change is it performing? maybe an html file being rewritten each time? [16:00:07] And it sure looks like dumps are being generated there, because my last puppet run produced this output: [16:00:09] https://www.irccloud.com/pastebin/RqbgMVuD/ [16:00:30] they aren't being generated there [16:00:47] 110% they are not. the nfs shares where they are written are dumpsdata100x [16:01:00] then files get rsynced to the clouddumps hosts. [16:01:44] there's probably a dvd.html file that was added in puppet and updated, and some old copy that is on the primary dumps nfs share that is getting rsynced over, overwriting the one in puppet [16:02:57] yeah, that looks right [16:03:36] So I should find that file and remove it? [16:03:44] the easy fix then is to remove the copy on the primary nfs share, and toss it, indeed [16:03:53] and then migt as well toss it on the secondary nfs share too [16:04:26] if you make a task we can make sure it's gone from any spare nfs share, so that if some host is swapped in to replace another, the problem doesn't return [16:04:36] ok [16:04:44] and assign it to Will? [16:05:06] I wouldn't assign it. tag it with dumps-generation and uh [16:05:23] btullis: what's the incoming column for the data engineering workboard? [16:05:28] or what's the project tag? [16:05:38] #data-platform-sre (incoming) [16:06:25] but you can let them know on slack/irc, both Virginia and Will, of this, so they have it on the list of ;things Data Engineering needs to handle if they come up', that sort of thing [16:06:56] I'm supposed to stop being the contact point now (though I might help behind the scenes for a while) [16:07:27] thank you both! [16:08:26] ...whereas I am a certified control freak :-) [16:08:50] A pleasure. [16:11:05] Is there some server badness happening? A moment ago, I got a "something went wrong" type page with the message "Original error: upstream connect error or disconnect/reset before headers. reset reason: overflow" [16:11:23] Now I just got a page where the HTML loaded, but none of the CSS or JS. [16:11:58] btullis: tags look right here? https://phabricator.wikimedia.org/T346165 [16:15:22] andrewbogott: yes thanks. Perfect.