[07:18:36] how long does a reimage with an OS upgrade take to be visible on PCC? [09:34:41] taavi: the usual way was manual (see https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet-diffs#How_to_update_the_compiler's_facts?_(e.g._INFO:_Unable_to_find_facts_for_host_conf2001.codfw.wmnet,_skipping) ) but I know that j.bond was working recently on PCC to simplify that. I'm not sure in which state it is right now. [10:39:35] hmm I'm being tortured by dpkg-source and the following error: dpkg-source: error: unrepresentable changes to source [10:39:46] any suggestions on how to find the offending file(s)? [10:41:07] vgutierrez: uh oh [10:41:16] i'd summon Emperor [10:41:23] they saved me the last time this happened to me [10:42:01] what's the current ritual required to summon Emperor ? [10:42:31] considering that's Friday it must be fricking expensive.. I bet that some HP ink is required [10:43:08] gladly I got the cheap replacement here.. some unicorn blood [10:43:20] vgutierrez: that likely means some changes were made to the package source code outside of the debian/ dir [10:45:13] hmmm [10:46:11] vgutierrez: https://wikitech.wikimedia.org/wiki/Orchestrator#Updating_orchestrator_packaging_to_a_new_upstream_version - my docs from the last time i ran into this [10:46:14] note point 6 [10:46:41] this assumes you're working on something that uses gbp or equivalent [10:46:48] I wonder if failing to push the version tag could trigger this as well [10:50:34] vgutierrez: i don't _think_ that would do it [10:52:44] vgutierrez: mmm it depends, on some setups git-buildpackage may use the information from the git tree to checkout files and build the package (such as the last git tag). A missing git tag may mean not the right files were checked out. But who knows :-) debian packaging is a mess [10:56:32] per the build log it's complaining about a doc/html/_static/ajax-loader.gif: binary file contents changed (and being a non text-based file it cannot be represented as a patch): [10:56:44] 10:38:17 dpkg-source: error: cannot represent change to doc/html/_static/ajax-loader.gif: binary file contents changed [10:56:45] 10:38:17 dpkg-source: error: add doc/html/_static/ajax-loader.gif in debian/source/include-binaries if you want to store the modified binary in the debian tarball [10:57:18] now this is kind of strange, still since these files should be part of the upstream releases, right? [10:57:58] otherwise the include-binaries is only needed for some special files in debian/* (e.g. some packages ship a Debian-specific PNG icon or similar) [10:58:55] so it sounds like something related to the tar release handling [11:00:47] is this limited to the CI-triggered build, or does it also apply to a local build? [11:01:21] happens in deneb as well [11:02:11] vgutierrez: did you try creating a new changelog entry? [11:02:35] it's there [11:02:45] the change triggering the error on CI is the new changelog entry [11:04:23] * Emperor twitches [11:05:34] Emperor: \o/ [11:06:06] let me rubuild the upstream-tar [11:06:08] sigh [11:06:34] stupid question - is the complained-of .gif in your build tree the same as in the orig.tar.gz (or equivalent)? dpkg-source seems to think not [11:09:48] (I have mostly moved to dgit for packages I maintain myself these days, which has its own oddities but IMO makes life rather easier, not least 'cos it checks that the thing you're tring to build matches what you have in git) [11:09:56] obviously I'm missing one step in the process [11:10:18] source tree hasn't been updated on the debian branch 😓 [11:23:55] fixed.. thanks for your input, it helped me sorting out what the issue was [11:27:16] \o/ [11:28:16] and fixed our doc about building varnish :) [11:28:26] double win :-) [11:36:55] <_joe_> Emperor: i didn't know dgit! [11:43:55] dgit is great, and even had quite good documentation :) [13:23:59] I guess that's what the D stands for then.. [20:35:01] inflatador: I see an outstanding diff on elasticsearch with puppet-merge, okay to take out? [20:36:48] jhathaway yeah, should be fine [20:37:01] thubs [20:39:10] jhathaway was it this patch or something else? https://gerrit.wikimedia.org/r/c/operations/puppet/+/757772 [20:39:43] Brian King: deployment-prep: add cergen config for elastic service (9ba40ca) [20:41:01] ah, that is labs/private, I many times forget it need a noop deployment on puppetmaster [20:47:33] really, it can block product puppet merges? [20:47:40] errr...production puppet merges? [20:48:20] no block, it just "we don't merge anything we don't know about" [20:49:54] there was a reason for the manual aproval, but cannot remember, I think wmcs did the work [20:50:08] being able to merge anything onto labs/private gives you full root to all of cloud vps, that's why it has similar safeguards to the main puppet repository [20:50:17] ^so that :-) [20:51:40] I understand that, just wondering about the blast radius. unmerged commits to labs/private aren't showing up in the diff when you try to puppet-merge in prod, right? [20:52:38] I belive they do, that's why you got asked :-) [20:52:51] well, that's...not ideal [20:53:11] I belive there is a flag to skip it [20:53:54] the right solution is to run puppet-merge after merging things to labs/private [20:54:02] yeah [20:54:38] there is "--labsprivate" although not "--production" :-) [20:55:34] iirc the script is smart enough that you don't even need to specify that [20:55:37] Anyone got time to walk me through this? I already left a patch dangling yesterday and I don't want to hold anyone up, nor do I want to accidentally nuke all the local commits on deployment-puppetmaster04.deployment-prep.eqiad.wmflabs [20:58:08] I'm happy to help [20:58:13] it is just "sudo puppet-merge --labsprivate" on deployment-puppetmaster04.deployment-prep.eqiad.wmflabs ? [20:58:22] no [20:58:24] just puppet-merge [20:58:32] on puppetmaster1001 [20:58:38] all `puppet-merge` actions need to be run on production puppetmasters [20:59:22] OK [21:00:17] "Fetching new commits from: https://gerrit.wikimedia.org/r/labs/private No changes to merge." [21:00:24] sounds like we are good? [21:01:07] j.hathaway already merged your commits, I think? [21:02:29] Yeah, must have. I guess I misinterpreted his comment about "taking out" above [21:02:58] yes I did, sorry if that wasn't clear [21:05:02] no problems. Thanks taavi et al for your patience!