[00:50:16] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10MW-1.40-notes (1.40.0-wmf.13; 2022-12-05), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T320518 (10colewhite) >>! In T320518#8455688, @Ladsgroup wrote: > It's in a much better s...
[01:38:59] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10MW-1.40-notes (1.40.0-wmf.13; 2022-12-05), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T320518 (10Jdlrobson) Hey @colewhite I need a week's worth of data to address a deeper is...
[02:04:05] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10MW-1.40-notes (1.40.0-wmf.13; 2022-12-05), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T320518 (10Legoktm)
[03:15:44] <wikibugs>	 10Beta-Cluster-Infrastructure: beta-prometheus.wmflabs.org 502 Bad Gateway - https://phabricator.wikimedia.org/T324695 (10TheresNoTime) 05Open→03Invalid Expected, see {T324782} for follow-up
[03:36:00] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10MW-1.40-notes (1.40.0-wmf.13; 2022-12-05), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T320518 (10Legoktm) The train was rolled back because of T324801#8455874.
[06:15:54] <bawolff>	 Is it a known thing that doing git fetch on http://gerrit.wikimedia.org/r/p/mediawiki/extensions.git/ (note http not https) gives a 403? I'm not opposed, but it seems like a breaking change that maybe should have been announced
[07:06:23] <legoktm>	 it was announced
[07:49:45] <wikibugs>	 (03PS1) 10Dom Walden: Run selenium tests for extensions against test2wiki. [integration/config] - 10https://gerrit.wikimedia.org/r/866548 (https://phabricator.wikimedia.org/T303739)
[07:51:51] <wikibugs>	 (03CR) 10Dom Walden: "WikibaseLexeme tests will just run against https://wikidata.beta.wmflabs.org." [integration/config] - 10https://gerrit.wikimedia.org/r/866548 (https://phabricator.wikimedia.org/T303739) (owner: 10Dom Walden)
[08:40:45] <wikibugs>	 (03CR) 10Hashar: Replace CI results table by Gerrit Check API (031 comment) [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/859083 (https://phabricator.wikimedia.org/T214068) (owner: 10Hashar)
[09:05:35] <wikibugs>	 (03PS2) 10Dom Walden: Run selenium tests for extensions against test2wiki. [integration/config] - 10https://gerrit.wikimedia.org/r/866548 (https://phabricator.wikimedia.org/T303739)
[09:07:08] <wikibugs>	 10Scap, 10SRE: Wrong umask when deploying from screen - https://phabricator.wikimedia.org/T200690 (10Tgr) >>! In T200690#8265133, @dancy wrote: > @Tgr Can you confirm that this is still a problem?  Probably not because these days you'd use `scap backport` which runs git commants as the deploy user, there isn't...
[09:11:22] <wikibugs>	 (03CR) 10Dom Walden: "OK, so I think I will create a new project without WikibaseLexeme and Wikistories and set it to only email me, so no one else is spammed w" [integration/config] - 10https://gerrit.wikimedia.org/r/866548 (https://phabricator.wikimedia.org/T303739) (owner: 10Dom Walden)
[09:18:27] <wikibugs>	 (03CR) 10Hashar: Boilerplate for QUnit testing (032 comments) [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/861486 (owner: 10Hashar)
[09:20:24] <wikibugs>	 (03CR) 10Hashar: Replace CI results table by Gerrit Check API (031 comment) [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/859083 (https://phabricator.wikimedia.org/T214068) (owner: 10Hashar)
[09:24:06] <wikibugs>	 (03PS17) 10Hashar: Replace CI results table by Gerrit Check API [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/859083 (https://phabricator.wikimedia.org/T214068)
[09:24:09] <wikibugs>	 (03PS7) 10Hashar: Add unit testing with QUnit [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/861486
[09:29:39] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10MW-1.40-notes (1.40.0-wmf.13; 2022-12-05), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T320518 (10hashar) The blocker had a patch which needs to be backported. I can run the tr...
[09:33:11] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10MW-1.40-notes (1.40.0-wmf.13; 2022-12-05), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T320518 (10Ladsgroup) I'm backporting it asap
[09:59:32] <hashar>	 thank you Amir1 !
[10:02:33] <Amir1>	 yw
[10:52:41] <wikibugs>	 (03PS1) 10Ilias Sarantopoulos: inference-services: add revscoring pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/866560 (https://phabricator.wikimedia.org/T323586)
[10:54:21] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] inference-services: add revscoring pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/866560 (https://phabricator.wikimedia.org/T323586) (owner: 10Ilias Sarantopoulos)
[11:05:13] <wikibugs>	 (03CR) 10AikoChou: inference-services: add revscoring pipelines (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/866560 (https://phabricator.wikimedia.org/T323586) (owner: 10Ilias Sarantopoulos)
[11:07:48] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10MW-1.40-notes (1.40.0-wmf.13; 2022-12-05), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T320518 (10Ladsgroup)
[11:11:11] <wikibugs>	 (03PS2) 10Ilias Sarantopoulos: inference-services: add revscoring pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/866560 (https://phabricator.wikimedia.org/T323586)
[11:11:31] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: inference-services: add revscoring pipelines (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/866560 (https://phabricator.wikimedia.org/T323586) (owner: 10Ilias Sarantopoulos)
[11:12:55] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] inference-services: add revscoring pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/866560 (https://phabricator.wikimedia.org/T323586) (owner: 10Ilias Sarantopoulos)
[11:28:51] <wikibugs>	 10Continuous-Integration-Config, 10Codex, 10Design-Systems-Team (Design-Systems-Sprint), 10Patch-For-Review: docs: Consider making the latest release branch the default for the live docs site - https://phabricator.wikimedia.org/T311097 (10bmartinezcalvo) >>! In T311097#8453147, @Volker_E wrote: > @bmartine...
[11:34:13] <wikibugs>	 (03PS3) 10Ilias Sarantopoulos: inference-services: add revscoring pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/866560 (https://phabricator.wikimedia.org/T323586)
[11:35:23] <wikibugs>	 (03PS1) 10Ilias Sarantopoulos: inference-services: add revscoring pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/866570 (https://phabricator.wikimedia.org/T323586)
[11:35:57] <wikibugs>	 (03CR) 10jenkins-bot: inference-services: add revscoring pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/866560 (https://phabricator.wikimedia.org/T323586) (owner: 10Ilias Sarantopoulos)
[11:36:54] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] inference-services: add revscoring pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/866570 (https://phabricator.wikimedia.org/T323586) (owner: 10Ilias Sarantopoulos)
[11:38:27] <wikibugs>	 (03PS2) 10Ilias Sarantopoulos: inference-services: add revscoring pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/866570 (https://phabricator.wikimedia.org/T323586)
[11:39:15] <wikibugs>	 (03PS1) 10Ilias Sarantopoulos: inference-services: add revscoring pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/866571 (https://phabricator.wikimedia.org/T323586)
[11:43:51] <wikibugs>	 (03Abandoned) 10Ilias Sarantopoulos: inference-services: add revscoring pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/866560 (https://phabricator.wikimedia.org/T323586) (owner: 10Ilias Sarantopoulos)
[11:44:33] <wikibugs>	 (03Abandoned) 10Ilias Sarantopoulos: inference-services: add revscoring pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/866571 (https://phabricator.wikimedia.org/T323586) (owner: 10Ilias Sarantopoulos)
[12:47:33] <wikibugs>	 10Release-Engineering-Team, 10Fresh, 10MediaWiki-Core-Tests, 10Browser-Tests, and 2 others: Fresh problem when running Selenium tests - https://phabricator.wikimedia.org/T313899 (10zeljkofilipin) 05In progress→03Open
[12:56:15] <wikibugs>	 (03PS1) 10Jbond: operations-puppet: refresh to pick up sorted_set for ruby 3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/866584
[13:03:52] <wikibugs>	 (03PS1) 10Jbond: operations-puppet-catalog-compiler: add support for CHANGE_PRIVATE [integration/config] - 10https://gerrit.wikimedia.org/r/866585 (https://phabricator.wikimedia.org/T265633)
[13:19:13] <hashar>	 jbond: good morning ;)
[13:19:22] <jbond>	 hi hashar 
[13:19:36] <hashar>	 I can't remember whether we said I should train you up so you can deploy those jenkins jobs updates
[13:19:50] <jbond>	 im happy to 
[13:19:51] <hashar>	 or whether you preferred to use us as deployment service 
[13:19:53] <hashar>	 ;D
[13:20:39] <hashar>	 I think you theorically already have the permissions
[13:21:09] <jbond>	 i dont have +2 on the intergrations/config repo but suspect ill have the othr permissions
[13:21:35] <hashar>	 for Jenkins that requires one to be in the LDAP group `ciadmin`
[13:22:15] <jbond>	 happy to add myself if oyu approve :)
[13:22:16] <hashar>	 which is probably already the case for most SRE since that is how folks trigger the puppet compiler from ./utils/pcc.py
[13:22:38] <hashar>	 and for integration/config well hmm
[13:22:40] <wikibugs>	 (03PS3) 10Dom Walden: Run selenium tests for extensions against test2wiki. [integration/config] - 10https://gerrit.wikimedia.org/r/866548 (https://phabricator.wikimedia.org/T303739)
[13:23:33] <hashar>	 looks like SRE can grant themselves all rights cause they are owner of all repos if I understand it properly
[13:23:35] <jbond>	 fyi ciadmin is not required for ./utils/pcc.py
[13:24:25] <wikibugs>	 (03PS1) 10Hashar: Review access change [integration/config] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/866474
[13:25:26] <wikibugs>	 (03PS2) 10Hashar: Grant SRE Code-Review:+2 [integration/config] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/866474
[13:25:39] <hashar>	 jbond: doesn't it requires the permission to trigger a job?
[13:26:14] <jbond>	 hashar: no, because i use it al the time and up untill 5 mins ago i was not in ciadmin :)
[13:27:14] <hashar>	 oh
[13:27:32] <hashar>	 anyone from `wmf` LDAP group can trigger a build (as well as `nda` and `wmde` members)
[13:27:40] <jbond>	 it uses the jenkins python lib but, looking at the could i think it probably just posts to https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/build?delay=0sec
[13:27:47] <jbond>	 ahh ok
[13:28:01] <jbond>	 then yes i have those permissions
[13:28:12] <hashar>	 ciadmin grants you a bunch of other privileges to brick the instance entirely
[13:28:20] <hashar>	 and the permission to configure jobs
[13:29:36] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "What is written on the can, that grants CR+2" [integration/config] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/866474 (owner: 10Hashar)
[13:29:41] <wikibugs>	 (03CR) 10Hashar: [V: 03+2 C: 03+2] Grant SRE Code-Review:+2 [integration/config] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/866474 (owner: 10Hashar)
[13:30:08] <hashar>	 not sure why it was not set previosuly, I guess I always assumed SRE to have the permission everywhere
[13:30:16] <jbond>	 adding myself to ciadmin dose give me +2
[13:30:23] <hashar>	 AH
[13:30:49] <jbond>	 i think that some have admin everywhere and we can all promote our self if needed 
[13:31:17] <jbond>	 but by default new sre's only get ops
[13:31:36] <jbond>	 and wmf
[13:32:23] <jbond>	 oh wait ciadmin is probably not needed for +2 i just saw your change
[13:32:28] <jbond>	 ill remove myself from that
[13:32:31] <wikibugs>	 (03PS1) 10Hashar: Revert "Grant SRE Code-Review:+2" [integration/config] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/866475
[13:32:42] <jbond>	 oops to late :/
[13:32:44] <jbond>	 hashar: 
[13:32:45] <hashar>	 jbond: you do need to be in the ciadmin LDAP group
[13:32:57] <hashar>	 which also gives the permission in Jenkins to change a job configuration
[13:33:02] <jbond>	 oh oj
[13:33:04] <jbond>	 ok
[13:33:15] * jbond stasy in ciadmin
[13:33:17] <hashar>	 so it is all magically happening
[13:33:28] <wikibugs>	 (03CR) 10Hashar: [V: 03+2 C: 03+2] Revert "Grant SRE Code-Review:+2" [integration/config] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/866475 (owner: 10Hashar)
[13:33:56] <hashar>	 I knew it was a bit suspicious to have no permissions in Gerrit ;)
[13:33:58] <jbond>	 i suspect it may have been this change ^^ that you just rverted that actully gave me +2 but we will see in a second i gusse
[13:34:32] <jbond>	 yep im back at +1 sorrcy about the confusion there
[13:34:38] <hashar>	 lol
[13:35:26] <hashar>	 there are really only a few things I hate in this job
[13:35:29] <hashar>	 cache invalidation for sure
[13:35:34] <hashar>	 naming thing is often annoying
[13:35:37] <hashar>	 AND ACCESS LISTS
[13:35:57] <jbond>	 slyngs is fixing all of those issues ;)
[13:36:06] <jbond>	 sorry the last issue only
[13:36:36] <slyngs>	 The two firsts are unfixable
[13:36:58] <hashar>	 the first for sure: don't use cache!
[13:37:53] <hashar>	 I have granted CR+2 rights to members of the integration group which includes ldap/ciadmin
[13:37:58] <hashar>	 so that should gives you +2 jbond
[13:38:04] <hashar>	 or there is some faulty cache somewhere
[13:38:38] <slyngs>	 The second one was fixed by a Danish software company years ago. They had "The big book of variable names", before you could use a new variable you had to consult the keeper of the book, who would either find a existing variable for you to use, or if he felt generous: Assign you a new variable
[13:38:56] <hashar>	 oh that is great!
[13:39:07] <jbond>	 lol the irony, i suspect there is some faulty cache somewhere.  i think gerrit has a cache of ldap useres for about an hour or something.  i have seen some wikitech article on this 
[13:39:39] <hashar>	 that might be it
[13:39:58] <hashar>	 and I don't think there is away to query the permissions a user has or   the set of permissions in effect on a given repo
[13:40:14] <jbond>	 ssh -p 29418 gerrit.wikimedia.org gerrit flush-caches --cache ldap_groups
[13:40:35] * hashar copies past
[13:40:44] <hashar>	 !log ssh -p 29418 gerrit.wikimedia.org gerrit flush-caches --cache ldap_groups
[13:40:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:40:53] <jbond>	 yes that fixed it
[13:40:58] <hashar>	 \o/
[13:41:00] <hashar>	 it was cache
[13:41:05] <jbond>	 :D hehe
[13:41:05] <hashar>	 at least it was not DNS this time
[13:41:15] <hashar>	 so with that you can CR+2
[13:41:21] <hashar>	 and well it works like on puppet
[13:41:27] <jbond>	 yes i can
[13:41:33] <wikibugs>	 (03CR) 10Jbond: [C: 03+2] operations-puppet-catalog-compiler: add support for CHANGE_PRIVATE [integration/config] - 10https://gerrit.wikimedia.org/r/866585 (https://phabricator.wikimedia.org/T265633) (owner: 10Jbond)
[13:41:38] <hashar>	 but a CR+2 triggers a bunch of CI jobs which will Submit the change for you 
[13:41:49] <hashar>	 and afaik we don't have permission to vote Verified +1  nor do we have permission to Submit
[13:41:54] <hashar>	 so we MUST pass through CI
[13:42:03] <hashar>	 which is fun cause if CI is broken then we can't fix it ;)
[13:42:07] <jbond>	 yes i just did a +2 and its doinf the gate+submit now
[13:42:14] <jbond>	 lol :S
[13:42:43] <hashar>	 for changes to jjb, there is a CI job which gives the diff of the jobs xml
[13:42:55] <hashar>	 aka https://integration.wikimedia.org/ci/job/integration-config-jjb-diff-docker/5151/console
[13:43:07] <hashar>	 and if the jjb yaml is wrong, some other job catch it and fails 
[13:43:19] <wikibugs>	 (03Merged) 10jenkins-bot: operations-puppet-catalog-compiler: add support for CHANGE_PRIVATE [integration/config] - 10https://gerrit.wikimedia.org/r/866585 (https://phabricator.wikimedia.org/T265633) (owner: 10Jbond)
[13:43:43] <jbond>	 ok the jjb has based and merged
[13:43:49] <hashar>	 for deployment it is done from our local machine using jenkins-job-builder which generates the XML jobs config locally and push them to Jenkins
[13:43:51] <jbond>	 (have not +2 the other one yet)
[13:44:03] <hashar>	 the full doc to run it is at https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder
[13:44:20] <hashar>	 but in practice you only need a jenkins jobs conifguration file that has your user and a api token
[13:44:45] <hashar>	 and then one can run the helper at the root of the integraiton/config repo:  `./jjb-update '<fnmatch for the job(s) you wanna update>'`
[13:45:04] <hashar>	 so typically I would go to the jjb diff job  https://integration.wikimedia.org/ci/job/integration-config-jjb-diff-docker/5151/console
[13:45:16] <jbond>	 ack let me skim the wiki page quickly
[13:45:18] <hashar>	 find the part that list the altered job
[13:45:27] <hashar>	 then copy paste and: `./jjb-update operations-puppet-catalog-compiler`
[13:45:38] <hashar>	 and that is how I have terminated my job position
[13:46:14] <jbond>	 :D
[13:47:03] <hashar>	 the doc looks correct
[13:47:24] <hashar>	 it miss mentionning the jjb-update helper at the front, but I think it is fine to give extra details on the wiki page
[13:48:03] <hashar>	 for updates to the dockerfiles that is more or less the same, +2 then run an helper from the root of the repo `./fab deploy_docker`.  It ssh to contint.wikimedia.org and run `docker-pkg` there
[13:51:04] <jbond>	 yes ill update the pag as i go
[13:53:50] <jbond>	 so looking at the doc they suggest using tox to upload i.e.
[13:53:51] <jbond>	 tox -e jenkins-jobs -- --conf jenkins_jobs.ini update jjb/ 'example-jenkins-job'
[13:54:01] <jbond>	 im gussing that is instead of jjb-update
[13:54:01] <jbond>	 ?
[13:55:17] <jbond>	 that seems to have worked well https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/build?delay=0sec
[13:55:27] <hashar>	 yeah
[13:55:48] <hashar>	 since I was fed up of writing that super long line which I have been entering keystroke by keystroke for years
[13:56:02] <hashar>	 I eventually went to add some kind of shell entrypoint which does the magic for us
[13:56:15] <hashar>	 which got us `./jjb-update`
[13:56:24] <jbond>	 ahh ok
[13:56:26] <hashar>	 and to generate the XML locally `./jjb-test`
[13:56:54] <hashar>	 you can look at the script source it is pretty much the same
[13:57:09] <hashar>	 maybe one day I will migrate that repo to scap
[13:57:32] <jbond>	 i added a "Deploying changes to dockerfiles" section to the above wiki
[14:00:06] <wikibugs>	 10Project-Admins, 10Content-Transform-Team: "Parsoid Tracking tasks" in Phabricator vs Content Transform Team - https://phabricator.wikimedia.org/T324679 (10Aklapper) Thanks! This is currently a subproject of Parsoid (a codebase) but the list is of a team. How to proceed? Should #Content-Transform-Team have so...
[14:01:38] <hashar>	 jbond: +1 )
[14:01:50] <jbond>	 ok i made a few other edits to https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder#Deploy_changes to add the helper scripts
[14:02:02] <jbond>	 sorry the are ananouymous wasn;t logged into meta
[14:02:22] <hashar>	 then on Jenkins, we have a plugin that keeps track of config changes as well
[14:02:33] <wikibugs>	 (03CR) 10Jbond: [C: 03+2] operations-puppet: refresh to pick up sorted_set for ruby 3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/866584 (owner: 10Jbond)
[14:02:38] <hashar>	 so on https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ , if you are a ciadmin, you can browse the job config history
[14:02:45] <hashar>	 (a link somewhere on the side bar)
[14:02:52] <hashar>	 which brings us to https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/jobConfigHistory/
[14:03:15] <hashar>	 and there is a somehow poor diff available https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/jobConfigHistory/showDiffFiles?timestamp1=2022-10-06_13-29-35&timestamp2=2022-12-09_13-54-54
[14:03:33] <hashar>	 then in practice the yaml diff and the output of our jenkins job that does the diff of xml config is sufficient
[14:03:39] <hashar>	 so well
[14:03:43] <hashar>	 congratulations :-]
[14:04:12] <jbond>	 jbond is missing the Job/Configure permission
[14:04:19] <wikibugs>	 (03Merged) 10jenkins-bot: operations-puppet: refresh to pick up sorted_set for ruby 3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/866584 (owner: 10Jbond)
[14:05:13] <hashar>	 but you have deployed the job!?
[14:05:47] <hashar>	 ah no it is an extra permission
[14:05:48] <hashar>	 fun
[14:06:21] <jbond>	 :)
[14:06:46] <jbond>	 !log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/866584
[14:06:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:07:08] <hashar>	 !log CI Jenkins: granted "Job Config History" permissions to members of `ciadmin` LDAP group
[14:07:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:08:16] <hashar>	 for the docker images, all the magic is setup on our two prod machines contint2001 (the primary) contint1002  (brand new as of yesterday)
[14:08:42] <hashar>	 the `./fab deploy_docker` ssh to `contint.wikimedia.org` which points to the current primary (contint2001`
[14:08:52] <hashar>	 then runs docker-pkg with some config file somewhere under /etc
[14:09:00] <hashar>	 that has the credentials to upload to the docker registry
[14:09:08] <hashar>	 no clue about the permissions
[14:09:43] <hashar>	 the file is owned by root:contint-admins
[14:09:43] <jbond>	 thats running now seems to be goind ok so far
[14:09:53] <hashar>	 the group comes from modules/admin/data...  not from LDAP
[14:10:13] <jbond>	 i guess it uses sudo?
[14:10:15] <hashar>	 so maybe as a normal user you might end up encountering a permission issue when uploading the built image
[14:10:28] <jbond>	 but yes ill keep a look out
[14:10:55] <hashar>	 in which case we will have to add you to the contint-admins group in puppet
[14:11:16] <jbond>	 ack ill let it run and see where/if it errors and send a follow up patch
[14:11:30] <hashar>	 that is until someone migrates our management of unix groups toward LDAP
[14:11:41] <jbond>	 however i think sly.ngs did this last week and i didn;t see an update to contint-admins so we wil see
[14:12:15] <hashar>	 yeah see above my 3rd point about ACCESS LISTS
[14:12:16] <hashar>	 .D
[14:12:17] <hashar>	 ;D
[14:12:50] <jbond>	 :D
[14:17:01] <wikibugs>	 (03PS1) 10Jaime Nuche: kubernetes: update K8s namespace name to match helm chart [tools/train-dev] - 10https://gerrit.wikimedia.org/r/866592
[14:17:03] <wikibugs>	 (03PS1) 10Jaime Nuche: run Kubernetes steps by default [tools/train-dev] - 10https://gerrit.wikimedia.org/r/866593
[14:29:48] <wikibugs>	 (03PS1) 10Jbond: jjb/operations-puppet.yaml: update to use latest docker imager [integration/config] - 10https://gerrit.wikimedia.org/r/866599
[14:38:26] <wikibugs>	 (03CR) 10Jbond: [C: 03+2] jjb/operations-puppet.yaml: update to use latest docker imager [integration/config] - 10https://gerrit.wikimedia.org/r/866599 (owner: 10Jbond)
[14:40:13] <wikibugs>	 (03Merged) 10jenkins-bot: jjb/operations-puppet.yaml: update to use latest docker imager [integration/config] - 10https://gerrit.wikimedia.org/r/866599 (owner: 10Jbond)
[15:10:28] <wikibugs>	 10Phabricator, 10Data-Persistence (work done), 10SRE, 10decommission-hardware, and 3 others: decommission phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T323418 (10Jclark-ctr)
[15:10:51] <wikibugs>	 10Phabricator, 10Data-Persistence (work done), 10SRE, 10decommission-hardware, and 3 others: decommission phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T323418 (10Jclark-ctr) 05Open→03Resolved
[15:10:57] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Jclark-ctr)
[15:31:01] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10MW-1.40-notes (1.40.0-wmf.13; 2022-12-05), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T320518 (10colewhite) >>! In T320518#8455841, @Jdlrobson wrote: > Hey @colewhite I need a...
[15:45:00] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Investigate dropping diff.h2.db and quota.repo_size.h2.db disk caches - https://phabricator.wikimedia.org/T324771 (10hashar) **diff.h2.db** has been removed by Gerrit 3.5.0 ( https://gerrit-review.googlesource.com/c/gerrit/+/318335 ).  **quota.repo_size** seems to originat...
[15:51:01] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10MediaWiki-Core-Tests, 10Code-Health, and 8 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10kostajh)
[16:17:41] <wikibugs>	 (03CR) 10Ahmon Dancy: [C: 03+2] run Kubernetes steps by default [tools/train-dev] - 10https://gerrit.wikimedia.org/r/866593 (owner: 10Jaime Nuche)
[16:18:24] <wikibugs>	 (03CR) 10Ahmon Dancy: [C: 03+2] "Tested in train-dev after rebuilding and updating the scap checkout in the workdir." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/866592 (owner: 10Jaime Nuche)
[16:19:27] <wikibugs>	 (03Merged) 10jenkins-bot: kubernetes: update K8s namespace name to match helm chart [tools/train-dev] - 10https://gerrit.wikimedia.org/r/866592 (owner: 10Jaime Nuche)
[16:19:29] <wikibugs>	 (03Merged) 10jenkins-bot: run Kubernetes steps by default [tools/train-dev] - 10https://gerrit.wikimedia.org/r/866593 (owner: 10Jaime Nuche)
[16:33:50] <wikibugs>	 (03PS2) 10Jforrester: dockerfiles: Drop xdebug everywhere, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/866388 (https://phabricator.wikimedia.org/T319495)
[16:34:08] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] dockerfiles: Drop xdebug everywhere, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/866388 (https://phabricator.wikimedia.org/T319495) (owner: 10Jforrester)
[16:36:01] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: Drop xdebug everywhere, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/866388 (https://phabricator.wikimedia.org/T319495) (owner: 10Jforrester)
[16:36:31] <James_F>	 !log Docker: Publishing CI images without the xdebug extension per T319495
[16:36:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:36:34] <stashbot>	 T319495: Drop xdebug from all RelEng images - https://phabricator.wikimedia.org/T319495
[16:37:36] <wikibugs>	 (03PS1) 10Ahmon Dancy: A few more mwdebug -> mw-debug changes [tools/train-dev] - 10https://gerrit.wikimedia.org/r/866615
[16:46:49] <brennen>	 !log temporarily disabling gitlab-phabricator push notifications
[16:46:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:47:02] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-Page-history: Edits are incorrectly marked as "updated since your last visit" in page history on betacommons - https://phabricator.wikimedia.org/T307173 (10Aklapper)
[16:51:30] <wikibugs>	 (03PS1) 10Jforrester: jjb: Update jobs to PHP images without xdebug [integration/config] - 10https://gerrit.wikimedia.org/r/866618 (https://phabricator.wikimedia.org/T319495)
[16:53:19] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+2] A few more mwdebug -> mw-debug changes [tools/train-dev] - 10https://gerrit.wikimedia.org/r/866615 (owner: 10Ahmon Dancy)
[16:53:54] <wikibugs>	 (03Merged) 10jenkins-bot: A few more mwdebug -> mw-debug changes [tools/train-dev] - 10https://gerrit.wikimedia.org/r/866615 (owner: 10Ahmon Dancy)
[16:54:23] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-extensions-Gadgets, 10Security-Team, 10Security: Allow Javascript files from Wikimedia GitLab to be loaded as scripts in Wikimedia wikis - https://phabricator.wikimedia.org/T321458 (10Bugreporter) We can host scripts at another domain like cdn.wikimedia.org. Gitlab h...