[01:42:17] <wikibugs>	 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10diegodlh) Hi! I'm the project manager of [[ https://meta.wikimedia.org/wiki/Web2Cit | Web2Cit ]].  The project has a couple subprojects (research and community) that I'd like to crea...
[01:52:51] <wikibugs>	 10Project-Admins: Create project tag for Web2Cit - https://phabricator.wikimedia.org/T292323 (10diegodlh) Hi @Aklapper. Sorry for re-opening this, but I thought it'd be better than opening a separate task (let me know if that's what I should do instead).  I don't seem to be able to manage the project (i.e., edit...
[07:39:28] <hashar>	 jelto: hi, for hwen you are around may we enable Gitlab CSP ( https://gerrit.wikimedia.org/r/c/operations/puppet/+/737968 ) :)
[07:39:58] <hashar>	 looks like it is just going to break some firefox extensions due to a bug in firefox
[07:42:30] <wikibugs>	 (03PS2) 10Hashar: jjb: avoid chaining docker-run* macros [integration/config] - 10https://gerrit.wikimedia.org/r/739350
[07:43:14] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "No changes!" [integration/config] - 10https://gerrit.wikimedia.org/r/739350 (owner: 10Hashar)
[07:45:26] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: avoid chaining docker-run* macros [integration/config] - 10https://gerrit.wikimedia.org/r/739350 (owner: 10Hashar)
[08:16:08] <wikibugs>	 (03Abandoned) 10Hashar: Shellbox: drop composer tests [integration/config] - 10https://gerrit.wikimedia.org/r/651167 (owner: 10Giuseppe Lavagetto)
[08:19:49] <wikibugs>	 (03PS2) 10Hashar: dockerfiles: init test repo with explicit branch name [integration/config] - 10https://gerrit.wikimedia.org/r/737679
[08:20:36] <wikibugs>	 (03CR) 10Hashar: dockerfiles: init test repo with explicit branch name (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/737679 (owner: 10Hashar)
[08:23:09] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] dockerfiles: init test repo with explicit branch name [integration/config] - 10https://gerrit.wikimedia.org/r/737679 (owner: 10Hashar)
[08:24:59] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: init test repo with explicit branch name [integration/config] - 10https://gerrit.wikimedia.org/r/737679 (owner: 10Hashar)
[08:26:00] <wikibugs>	 (03PS4) 10Hashar: jjb: add the template name to the job description [integration/config] - 10https://gerrit.wikimedia.org/r/737723
[08:26:25] <hashar>	 !log Refreshing Jenkins jobs description to include the JJB template name in their description:  https://gerrit.wikimedia.org/r/c/integration/config/+/737723 
[08:26:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:27:12] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] jjb: add the template name to the job description [integration/config] - 10https://gerrit.wikimedia.org/r/737723 (owner: 10Hashar)
[08:28:59] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: add the template name to the job description [integration/config] - 10https://gerrit.wikimedia.org/r/737723 (owner: 10Hashar)
[08:48:21] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] "I have looked at all images and here is the result:" [integration/config] - 10https://gerrit.wikimedia.org/r/694621 (https://phabricator.wikimedia.org/T280170) (owner: 10Daimona Eaytoy)
[10:22:47] <wikibugs>	 (03PS1) 10Hashar: jjb: remove docker-run-with-log macro [integration/config] - 10https://gerrit.wikimedia.org/r/739483
[10:23:03] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] jjb: remove docker-run-with-log macro [integration/config] - 10https://gerrit.wikimedia.org/r/739483 (owner: 10Hashar)
[10:24:45] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: remove docker-run-with-log macro [integration/config] - 10https://gerrit.wikimedia.org/r/739483 (owner: 10Hashar)
[10:30:31] <wikibugs>	 (03PS1) 10Hashar: utils/jjb-diff: properly remove all temp directories [integration/config] - 10https://gerrit.wikimedia.org/r/739484
[10:31:17] <wikibugs>	 (03CR) 10Hashar: "Found that after I repeatedly ran utils/jjb-diff.sh, my temporary directory was completely filed with parent change configs :D" [integration/config] - 10https://gerrit.wikimedia.org/r/739484 (owner: 10Hashar)
[10:42:41] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10SRE: find a way to systematically update the deployment server name across all repos - https://phabricator.wikimedia.org/T197470 (10hnowlan) >>! In T197470#7507203, @dancy wrote: >>>! In T197470#7506161, @hnowlan wrote: >> For the immediate term if t...
[10:50:04] <wikibugs>	 10Project-Admins, 10User-dcaro: Create tag projects worktype-project, origin-user, origin-alert, origin-team - https://phabricator.wikimedia.org/T295692 (10dcaro) Thanks for the example!  >  I would generally avoid Subprojects  Yep, specially the fact that you can only have one of this tags per parent project...
[11:05:05] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10Zuul, 10observability, 10Patch-For-Review: Migrate Zuul alerting to Grafana / AlertManager - https://phabricator.wikimedia.org/T292284 (10hashar) 05Open→03Resolved @BTullis gave me a few more explanations about the various...
[11:19:46] <wikibugs>	 (03PS1) 10Hashar: jjb: normalize jobs to have /src on the host [integration/config] - 10https://gerrit.wikimedia.org/r/739515 (https://phabricator.wikimedia.org/T292729)
[11:27:16] <wikibugs>	 (03PS1) 10Hashar: jjb: normalize Quibble jobs to use /src [integration/config] - 10https://gerrit.wikimedia.org/r/739517
[11:27:18] <wikibugs>	 (03PS1) 10Hashar: jjb: remove docker-run-with-log-and-workspace-cache macro [integration/config] - 10https://gerrit.wikimedia.org/r/739518
[12:49:51] <wikibugs>	 10Project-Admins: Create project tag for Web2Cit - https://phabricator.wikimedia.org/T292323 (10Aklapper) @diegodlh: I've added you to #Trusted-Contributors, please try again
[12:51:01] <wikibugs>	 (03CR) 10Hashar: "recheck after updating integration-config* jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/739515 (https://phabricator.wikimedia.org/T292729) (owner: 10Hashar)
[13:12:36] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] jjb: normalize jobs to have /src on the host [integration/config] - 10https://gerrit.wikimedia.org/r/739515 (https://phabricator.wikimedia.org/T292729) (owner: 10Hashar)
[13:14:46] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: normalize jobs to have /src on the host [integration/config] - 10https://gerrit.wikimedia.org/r/739515 (https://phabricator.wikimedia.org/T292729) (owner: 10Hashar)
[13:23:17] <wikibugs>	 (03PS2) 10Hashar: jjb: normalize Quibble jobs to use /src [integration/config] - 10https://gerrit.wikimedia.org/r/739517 (https://phabricator.wikimedia.org/T292729)
[13:27:10] <hashar>	 !log Updating Quibble jobs with `docker run --volume "$(pwd)/src:/workspace/src"`  https://gerrit.wikimedia.org/r/c/integration/config/+/739517
[13:27:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:30:40] <hashar>	 if Quibble jobs have an issue, well I guess it meant I have screwed up https://gerrit.wikimedia.org/r/c/integration/config/+/739517 :D
[13:33:00] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] jjb: normalize Quibble jobs to use /src [integration/config] - 10https://gerrit.wikimedia.org/r/739517 (https://phabricator.wikimedia.org/T292729) (owner: 10Hashar)
[13:33:09] <wikibugs>	 (03PS2) 10Hashar: jjb: remove docker-run-with-log-and-workspace-cache macro [integration/config] - 10https://gerrit.wikimedia.org/r/739518
[13:33:13] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] jjb: remove docker-run-with-log-and-workspace-cache macro [integration/config] - 10https://gerrit.wikimedia.org/r/739518 (owner: 10Hashar)
[13:34:58] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: normalize Quibble jobs to use /src [integration/config] - 10https://gerrit.wikimedia.org/r/739517 (https://phabricator.wikimedia.org/T292729) (owner: 10Hashar)
[13:35:01] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: remove docker-run-with-log-and-workspace-cache macro [integration/config] - 10https://gerrit.wikimedia.org/r/739518 (owner: 10Hashar)
[14:19:17] <majavah>	 James_F: not sure if you noticed, but deployment-docker-wikifunctions01 seems to be failing to run puppet
[14:37:46] <wikibugs>	 (03PS1) 10JMeybohm: helm_linter: Ensure not helm repositories are defined [integration/config] - 10https://gerrit.wikimedia.org/r/739544
[14:37:48] <wikibugs>	 (03PS1) 10JMeybohm: jjb: update helm-linter job to releng/helm-linter:0.2.18 [integration/config] - 10https://gerrit.wikimedia.org/r/739545
[14:58:42] <wikibugs>	 10Phabricator, 10WMF-Legal: Phabricator Macros and other media are to have clear copyright status - https://phabricator.wikimedia.org/T128771 (10valerio.bozzolan) BTW the spirit of macros is to have memes, for addition, not in high quality. Simple graphics and memes in most countries do not fall under copyrigh...
[15:04:05] <wikibugs>	 (03PS2) 10Hashar: jjb: play with Jinja2 [integration/config] - 10https://gerrit.wikimedia.org/r/739282
[15:26:33] <wikibugs>	 (03CR) 10Daimona Eaytoy: dockerfiles: Install pcov in base PHP images (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/694621 (https://phabricator.wikimedia.org/T280170) (owner: 10Daimona Eaytoy)
[15:44:24] <James_F>	 majavah: Yeah, it won't let me ssh in for some reason.
[15:46:03] <majavah>	 James_F: try now?
[15:46:20] <James_F>	 majavah: Huh, yeah, I made it in this time; what did you do?
[15:46:28] <James_F>	 (Thanks!)
[15:46:40] <wikibugs>	 (03CR) 10Legoktm: dockerfiles: Install pcov in base PHP images (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/694621 (https://phabricator.wikimedia.org/T280170) (owner: 10Daimona Eaytoy)
[15:47:17] <majavah>	 I manually changed "admin" to "deployment-prep" in /etc/security/access.conf, usually puppet does that but can't if puppet fails
[15:47:59] <James_F>	 Aha.
[15:48:05] <James_F>	 Yeah, the initial puppet run hasn't.
[15:49:42] <James_F>	 OK, I've set the custom puppetmaster.
[15:49:45] <James_F>	 Now waiting for the cron.
[15:52:45] <bd808>	 You can force a puppet run with `sudo -i puppet agent -tv` if that's the cron you are waiting for
[15:53:02] <James_F>	 bd808: Not if I'm not yet in the sudoers list I can't.
[15:53:18] <bd808>	 heh, well there is that :)
[15:54:01] <wikibugs>	 (03PS3) 10Hashar: jjb: play with Jinja2 [integration/config] - 10https://gerrit.wikimedia.org/r/739282
[15:54:51] <bd808>	 !log Forced puppet run on deployment-docker-wikifunctions01.deployment-prep.eqiad1.wikimedia.cloud
[15:54:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:54:57] <James_F>	 bd808: Aww, thanks.
[15:54:57] <bd808>	 James_F: "Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Function lookup() did not find a value for the name 'profile::docker::engine::settings' (file: /etc/puppet/modules/profile/manifests/docker/engine.pp, line: 6) on node deployment-docker-wikifunctions01.deployment-prep.eqiad1.wikimedia.cloud"
[15:55:03] <James_F>	 Oh, hmm.
[15:55:20] <bd808>	 missing hiera in there somewhere
[15:55:52] <James_F>	 Yeah, it's just blank. I perhaps-foolishly assumed that it'd default to sensible things.
[15:56:08] <James_F>	 Given it's set to `profile::docker::engine::settings: {}` on the citoid box.
[15:56:36] <mutante>	   2 profile::docker::storage::physical_volumes: "/dev/md1"
[15:56:36] <mutante>	   3 profile::docker::storage::vg_to_remove: 'todelete'
[15:56:36] <mutante>	   4 profile::docker::engine::version: "18.06.3~ce~3-0~debian"
[15:56:36] <mutante>	   5 profile::docker::engine::packagename: "docker-ce"
[15:56:36] <mutante>	   6 profile::docker::engine::declare_service: true
[15:56:49] <bd808>	 it probably does, but only in production. a lot of these profiles have settings in a role that is not seen by the cloud vps puppetmasters
[15:56:49] <James_F>	 Thanks, I'll put all of those in.
[15:56:55] <mutante>	 you can copy from hieradata/role/common/kubernetes/worker.yaml
[15:56:56] <James_F>	 Right.
[15:57:10] <mutante>	 i would put it in the actual repo instead of using Horizon Hiera
[15:57:14] <mutante>	 unless its temp
[15:57:35] <mutante>	 both would work but if its in the repo you can forget about it in the future
[15:57:38] <James_F>	 It's temp until I know it works, then I can properly commit it.
[15:57:46] <bd808>	 mutante: you would, but those of us who are not prod roots likely would never do that
[15:57:52] <James_F>	 Also that.
[15:58:05] <bd808>	 making the config inaccessible to the project maintainers is not ideal
[15:58:20] <majavah>	 indeed, and it's currently a mess
[15:58:28] <James_F>	 bd808: Try now?
[15:58:57] <bd808>	 progress! "Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Function lookup() did not find a value for the name 'profile::docker::runner::service_defs' (file: /etc/puppet/modules/profile/manifests/docker/runner.pp, line: 24) on node deployment-docker-wikifunctions01.deployment-prep.eqiad1.wikimedia.cloud"
[15:59:24] <mutante>	 bd808: understood. all I can say is "add me on Gerrit, no IRC ping needed". best I have 
[15:59:46] <James_F>	 bd808: Try now?
[15:59:51] * James_F is very needy, sorry.
[16:00:29] <bd808>	 !log Forced puppet run on deployment-docker-wikifunctions01.deployment-prep.eqiad1.wikimedia.cloud after James_F added missing hiera bits
[16:00:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:00:45] <bd808>	 James_F: it completed. see if you have your sudo now?
[16:01:06] <bd808>	 doh. now it's in the puppetmaster switch state
[16:01:08] <James_F>	 Hmm, still prompting for password.
[16:01:14] <James_F>	 Oh, right, it's half-done?
[16:01:18] <bd808>	 yeah
[16:01:41] <majavah>	 try logging out and back in?
[16:01:46] <bd808>	 but I think that should have done the sudo stuff? you may need to log out and back in?
[16:02:21] <bd808>	 I'm actually sudo'ing there as my moral user and not root too which makes me think it should work...
[16:02:38] * James_F tries.
[16:02:55] <James_F>	 Apparently not.
[16:03:59] <wikibugs>	 10Project-Admins, 10User-dcaro: Create tag projects worktype-project, origin-user, origin-alert, origin-team - https://phabricator.wikimedia.org/T295692 (10MBinder_WMF) Given what's discussed so far (meaning there might be more worth considering)...probably? I can't say I have yet seen what you are describing....
[16:05:57] <majavah>	 /etc/sssd/sssd.conf still has ldap_sudo_search_base = ou=sudoers,cn=admin,ou=projects,dc=wikimedia,dc=org
[16:06:05] <majavah>	 cn=admin should have cn=deployment-prep instead
[16:06:31] <majavah>	 let me get its puppet certs sorted with the local puppetmaster to see if that helps
[16:07:27] <majavah>	 James_F: try now?
[16:07:44] <James_F>	 Yup, I'm a real boy now.
[16:07:45] <James_F>	 Thanks.
[16:09:28] <James_F>	 `E: Package 'docker-ce' has no installation candidate`
[16:09:40] <James_F>	 I guess I shouldn't have assumed that we can build on bullseye yet.
[16:10:23] <majavah>	 hmm
[16:11:40] <majavah>	 that probably should use docker.io, I think it comes from the instance hiera
[16:11:58] <majavah>	 docker-ce would be from docker's own repos, while docker.io is from debian upstream
[16:12:16] <majavah>	 and on buster and newer the one on debian is usually new enough
[16:13:40] <James_F>	 https://packages.debian.org/bullseye/docker.io 
[16:14:05] <James_F>	 But https://gerrit.wikimedia.org/g/operations/puppet/+/production/hieradata/role/common/kubernetes/worker.yaml#5 has docker-ce, I guess as a local package?
[16:15:41] <majavah>	 those nodes are on stretch
[16:15:44] <James_F>	 OK, switched to docker.io 20.10.5+dfsg1-1 now.
[16:16:52] <James_F>	 I guess the next step is to make our helm charts. Yay.
[16:17:46] <wikibugs>	 (03PS1) 10Zfilipin: jjb: Create selenium-daily-beta-VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/739565 (https://phabricator.wikimedia.org/T295362)
[16:33:43] <wikibugs>	 (03CR) 10Esther Akinloose: [C: 03+1] "Looks good." [integration/config] - 10https://gerrit.wikimedia.org/r/739565 (https://phabricator.wikimedia.org/T295362) (owner: 10Zfilipin)
[16:35:37] <wikibugs>	 (03CR) 10Jforrester: "❤️" [integration/config] - 10https://gerrit.wikimedia.org/r/739350 (owner: 10Hashar)
[16:37:57] <wikibugs>	 (03CR) 10Zfilipin: "The job is deployed and working: https://integration.wikimedia.org/ci/view/selenium-daily/job/selenium-daily-beta-VisualEditor/" [integration/config] - 10https://gerrit.wikimedia.org/r/739565 (https://phabricator.wikimedia.org/T295362) (owner: 10Zfilipin)
[16:38:26] <wikibugs>	 10Project-Admins, 10User-Urbanecm: Create tag for server-side upload requests - https://phabricator.wikimedia.org/T295231 (10Urbanecm) Filled {T295908} for video2commons.
[17:30:19] <wikibugs>	 (03PS4) 10Hashar: jjb: play with Jinja2 [integration/config] - 10https://gerrit.wikimedia.org/r/739282
[17:46:32] <wikibugs>	 (03PS1) 10Hashar: jjb: more migration to yaml based volumes definitions [integration/config] - 10https://gerrit.wikimedia.org/r/739583
[17:46:56] <wikibugs>	 (03CR) 10Hashar: [C: 04-2] "-2 cause some volumes magically disappear!" [integration/config] - 10https://gerrit.wikimedia.org/r/739583 (owner: 10Hashar)
[17:49:53] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] "Some volumes are missing :(" [integration/config] - 10https://gerrit.wikimedia.org/r/739282 (owner: 10Hashar)
[17:55:16] <wikibugs>	 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10GitLab (Auth & Access), 10Upstream, 10User-brennen: Gitlab 2fa password validation seems bugged - https://phabricator.wikimedia.org/T292431 (10mpopov) Thank you! Can also confirm this works (added 2FA and Yubikey to my account)
[18:03:05] <hashar>	 James_F: I found out earlier that utils/jjb-diff.sh leaves the parent XML files in the tmp dir and I managed to file my 2G tmpfs :D  https://gerrit.wikimedia.org/r/c/integration/config/+/739484 
[18:03:11] <hashar>	 seems to work for me
[18:18:38] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] utils/jjb-diff: properly remove all temp directories [integration/config] - 10https://gerrit.wikimedia.org/r/739484 (owner: 10Hashar)
[18:18:46] <hashar>	 thx ;)
[18:18:50] <James_F>	 hashar: Of course.
[18:23:15] <wikibugs>	 (03Merged) 10jenkins-bot: utils/jjb-diff: properly remove all temp directories [integration/config] - 10https://gerrit.wikimedia.org/r/739484 (owner: 10Hashar)
[18:43:17] <hashar>	 brennen: https://meta.wikimedia.org/wiki/IRC/Instructions#New_channel_setup :D
[18:46:15] <hashar>	 brennen: and my notes at https://phabricator.wikimedia.org/T286770#7457672
[18:47:26] <brennen>	 thanks hashar!
[19:05:35] <hashar>	 I cant find those VMs :\
[19:07:01] <hashar>	 found!
[19:10:14] <wikibugs>	 (03CR) 1020after4: [C: 03+2] "lgtm" [tools/scap] - 10https://gerrit.wikimedia.org/r/739040 (https://phabricator.wikimedia.org/T295304) (owner: 10Ahmon Dancy)
[19:11:17] <wikibugs>	 (03Merged) 10jenkins-bot: sync_master: Use same CDB rebuild technique as main deploy server [tools/scap] - 10https://gerrit.wikimedia.org/r/739040 (https://phabricator.wikimedia.org/T295304) (owner: 10Ahmon Dancy)
[19:13:08] <hashar>	 brennen: the Ganeti harware I have mentioned during the meeting, I have commented on the task. But they probably are no more dedicated
[19:13:22] <brennen>	 thanks for following up
[19:13:28] <wikibugs>	 10Release-Engineering-Team (Radar), 10GitLab, 10Security-Team, 10serviceops, 10SecTeam-Processed: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10hashar) After talking with @Brennen today, a while back we got budget to add some hardware to the Ganeti cluster. The...
[19:13:37] <hashar>	 for WMCS well we gotta ask :)
[19:14:04] <hashar>	 and I guess experiment digital ocean vs google cloud engine vs aws vs our own openstack
[19:14:09] <hashar>	 or something that 
[19:14:45] <hashar>	 well I am off
[19:14:52] <wikibugs>	 10Release-Engineering-Team, 10GitLab (Administration, Settings & Policy), 10User-brennen: Create an IRC channel for GitLab collaboration - https://phabricator.wikimedia.org/T295917 (10brennen)
[19:14:58] <hashar>	 Jinja2 is not pleasant at all :-\
[19:15:56] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10GitLab (Administration, Settings & Policy), 10User-brennen: Create an IRC channel for GitLab collaboration - https://phabricator.wikimedia.org/T295917 (10brennen)
[19:15:59] <twentyafterfour>	 hashar: I have yet to meet a template language that's pleasant 
[19:16:27] <twentyafterfour>	 the best you can hope for is minimizing the pain 
[19:17:04] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10GitLab (Administration, Settings & Policy), 10User-brennen: Create an IRC channel for GitLab collaboration - https://phabricator.wikimedia.org/T295917 (10hashar)
[19:17:06] <hashar>	 https://dhall-lang.org/ !
[19:17:27] <hashar>	 it is a turing complete template lagnuage to generate templates :D
[19:18:43] <hashar>	 brennen: for the IRC bots there are a bunch of configuration patch on https://phabricator.wikimedia.org/T286770#7457672  and following comments
[19:18:59] <hashar>	 and https://meta.wikimedia.org/wiki/Wm-bot for logging
[19:19:39] <hashar>	 overall #wikimedia-ops people are super helpful (our fleet of IRC channels is maintained by them)
[19:20:03] <hashar>	 twentyafterfour: https://dhall-lang.org/  is a nice thing. I have yet to find time to play with it though
[19:20:07] <hashar>	 I am off &&
[19:20:52] <brennen>	 sweet, thanks
[19:20:59] <brennen>	 have a good one!
[19:24:49] <wikibugs>	 (03CR) 10Ahmon Dancy: sync_master: Use same CDB rebuild technique as main deploy server (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/739040 (https://phabricator.wikimedia.org/T295304) (owner: 10Ahmon Dancy)
[19:41:05] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Maps, 10Product-Infrastructure-Team-Backlog: Puppet config is broken for the maps instance on deployment-prep - https://phabricator.wikimedia.org/T291624 (10MSantos) p:05Triage→03Low
[19:48:04] <dancy>	 !log upgrading scap in beta
[19:48:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:58:27] <wikibugs>	 10Release-Engineering-Team, 10Security-Team: Determine a batter way to track Wikimedia production security patches - https://phabricator.wikimedia.org/T295925 (10sbassett)
[20:00:34] <wikibugs>	 10Release-Engineering-Team, 10Security-Team: Determine a batter way to track Wikimedia production security patches - https://phabricator.wikimedia.org/T295925 (10sbassett)
[20:01:05] <wikibugs>	 10Release-Engineering-Team, 10Security-Team: Determine a batter way to track Wikimedia production security patches - https://phabricator.wikimedia.org/T295925 (10sbassett)
[20:04:57] <wikibugs>	 10Project-Admins: Create project tag for Web2Cit - https://phabricator.wikimedia.org/T292323 (10diegodlh) @Aklapper, it works now. Thank you! If you could consider adding me Project-Admins too so I can create the subprojects needed, as requested in T706#7508755, I'd appreciate it.  Thank you!
[20:07:46] <wmf-insecte>	 Project beta-scap-sync-world build #27728: 04FAILURE in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/27728/
[20:08:34] <dancy>	 ^ me
[20:20:09] <wmf-insecte>	 Project beta-update-databases-eqiad build #54676: 04FAILURE in 9.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/54676/
[20:20:52] <dancy>	 ^ I don't think that one is me.
[20:21:46] <wikibugs>	 10Release-Engineering-Team, 10Security-Team: Determine a better way to track Wikimedia production security patches - https://phabricator.wikimedia.org/T295925 (10brennen)
[20:40:06] <wikibugs>	 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Write and send release announcement for 1.37.0 - https://phabricator.wikimedia.org/T289594 (10Reedy)
[20:45:29] <Reedy>	 Error 1054: Unknown column 'ts_title' in 'translate_stash'
[20:45:55] <Reedy>	 I2290d582d6d315c353a0d9a03ec863dcaff34339
[20:46:31] <Reedy>	 Commented on the gerrit changeset
[20:52:11] <wikibugs>	 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T293950 (10jeena)
[20:52:57] <wikibugs>	 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Write and send release announcement for 1.37.0 - https://phabricator.wikimedia.org/T289594 (10Jdforrester-WMF)
[20:53:13] <wmf-insecte>	 Yippee, build fixed!
[20:53:13] <wmf-insecte>	 Project beta-scap-sync-world build #27729: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/27729/
[21:03:27] <wikibugs>	 (03PS5) 10Hashar: jjb: play with Jinja2 [integration/config] - 10https://gerrit.wikimedia.org/r/739282
[21:06:21] <wikibugs>	 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T293950 (10jeena)
[21:20:11] <wmf-insecte>	 Project beta-update-databases-eqiad build #54677: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/54677/
[21:37:04] <wikibugs>	 10Phabricator: Editing a maniphest form has wrong preamble - https://phabricator.wikimedia.org/T295934 (10Urbanecm)
[21:49:11] <wikibugs>	 10Phabricator, 10WMF-Legal: Phabricator Macros and other media are to have clear copyright status - https://phabricator.wikimedia.org/T128771 (10AntiCompositeNumber) For what its worth, the [[ https://foundation.wikimedia.org/wiki/Terms_of_Use/Phabricator | Phabricator terms of use ]] (linked in the footer) in...
[21:49:37] <wikibugs>	 (03PS1) 10Hashar: jjb: don't mount src twice [integration/config] - 10https://gerrit.wikimedia.org/r/739654
[21:58:09] <wikibugs>	 (03PS2) 10Hashar: jjb: don't mount src twice [integration/config] - 10https://gerrit.wikimedia.org/r/739654
[22:02:57] <hashar>	 it is definitely allll tooo messy
[22:10:24] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #29650: 04FAILURE in 5 min 57 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/29650/
[22:20:09] <wmf-insecte>	 Project beta-update-databases-eqiad build #54678: 04STILL FAILING in 8.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/54678/
[22:21:44] <urbanecm>	 ^^checking^^
[22:22:11] <Reedy>	 Still the same failure
[22:22:31] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash, 10observability, 10SRE Observability (FY2021/2022-Q2): Logstash in beta fails periodically - https://phabricator.wikimedia.org/T211984 (10colewhite) 05Open→03Resolved
[22:22:51] <wikibugs>	 10Phabricator, 10WMF-Legal: Phabricator Macros and other media are to have clear copyright status - https://phabricator.wikimedia.org/T128771 (10Dzahn) list of macros I created:  [[ https://phabricator.wikimedia.org/macro/view/19/ | bikeshed ]] - By Wester, CC BY-SA 2.0 BE, via Wikimedia Commons [[ https://pha...
[22:23:16] * urbanecm thought it's not known
[22:23:47] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): beta logstash servers run out of disk space - https://phabricator.wikimedia.org/T288989 (10colewhite) 05Open→03Resolved a:03colewhite Boldly resolving because we have migrated to a new [[ https://beta-logs.wmcloud.org | cluster ]] serving...
[22:23:51] <wikibugs>	 10Beta-Cluster-Infrastructure, 10SRE Observability, 10Wikimedia-Logstash, 10observability: logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10colewhite)
[22:25:52] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Doing), 10Discovery-Search, 10Wikimedia-Logstash, 10observability: scap on beta fails canary check: KeyError: 'aggregations' - https://phabricator.wikimedia.org/T260667 (10colewhite) Since we moved to the new beta logging cluster, is this issue s...
[22:26:25] <Reedy>	 srsly
[22:26:26] <Reedy>	 https://github.com/wikimedia/mediawiki-extensions-Translate/commit/1e1b349106181ee34a63adb1c333817b31330998
[22:26:32] <Reedy>	 that was never deployed it seems
[22:26:41] <wikibugs>	 10Beta-Cluster-Infrastructure, 10SRE Observability, 10Wikimedia-Logstash, 10observability: [_field_stats] endpoint is deprecated! Use [_field_caps] instead or run a min/max aggregations on the desired fields. - https://phabricator.wikimedia.org/T241485 (10colewhite) 05Open→03Resolved a:03colewhite I...
[22:27:11] <urbanecm>	 Reedy: that...doesn't look like a failproof way to do migrations
[22:27:21] <Reedy>	 lol
[22:27:31] <Reedy>	 !log deployment-prep drop old commonswiki.translate_stash table
[22:27:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[22:29:13] <urbanecm>	 Reedy: you'll need to do it for more wikis though
[22:29:18] <Reedy>	 Maybe, maybe not
[22:29:19] <urbanecm>	 like metawiki https://www.irccloud.com/pastebin/bLPlOMD8/
[22:29:46] <wmf-insecte>	 Project beta-update-databases-eqiad build #54679: 04STILL FAILING in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/54679/
[22:30:24] <Reedy>	 !log deployment-prep drop old metawiki.translate_stash table
[22:30:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[22:31:58] <wmf-insecte>	 Project beta-update-databases-eqiad build #54680: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/54680/
[22:32:04] <urbanecm>	 !log deployment-prep drop old testwiki.translate_stash table
[22:32:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[22:32:35] <urbanecm>	 !log deployment-prep drop old wikidatawiki.translate_stash table
[22:32:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[22:33:40] <urbanecm>	 Reedy: do we know where did that column go? also interesting beta foundationwiki had it.
[22:33:48] <Reedy>	 did it go?
[22:33:54] <Reedy>	 Schema chang was never applied
[22:34:04] <Reedy>	 so will depend how old the beta wiki is
[22:34:12] <wmf-insecte>	 Yippee, build fixed!
[22:34:13] <wmf-insecte>	 Project beta-update-databases-eqiad build #54681: 09FIXED in 1 min 27 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/54681/
[22:34:23] <urbanecm>	 at least it's fixed now
[22:37:46] <wikibugs>	 10Release-Engineering-Team (Radar), 10GitLab, 10Security-Team, 10serviceops, 10SecTeam-Processed: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Dzahn) @Jelto I [[ https://wikitech.wikimedia.org/wiki/Ganeti#Verify_cluster_resource_availability | checked capacity...
[22:49:58] <wikibugs>	 10Project-Admins, 10User-Urbanecm: Create tag for server-side upload requests - https://phabricator.wikimedia.org/T295231 (10Aklapper)
[23:08:08] <wikibugs>	 10Phabricator: Editing a maniphest form has wrong preamble - https://phabricator.wikimedia.org/T295934 (10Aklapper) I can confirm this problem. Somehow https://phabricator.wikimedia.org/transactions/editengine/transactions.editengine.config/view/106/ does not exist.
[23:09:36] <wikibugs>	 10Phabricator: Editing a maniphest form has wrong preamble - https://phabricator.wikimedia.org/T295934 (10Aklapper) I cannot even find where that `IMPORTANT: Please remember, no donor PII` string comes from; doesn't seem to be the preamble itself either
[23:15:58] <wikibugs>	 (03PS1) 10Ahmon Dancy: get_active_wikiversions: Sort wikiversions keys before scanning [tools/scap] - 10https://gerrit.wikimedia.org/r/739671
[23:16:53] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] get_active_wikiversions: Sort wikiversions keys before scanning [tools/scap] - 10https://gerrit.wikimedia.org/r/739671 (owner: 10Ahmon Dancy)
[23:16:57] <wikibugs>	 (03PS2) 10Ahmon Dancy: get_active_wikiversions: Sort wikiversions keys before scanning [tools/scap] - 10https://gerrit.wikimedia.org/r/739671
[23:18:11] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10ci-test-error (WMF-deployed Build Failure): TAR_ENTRY_ERROR ENOSPC: no space left on device - https://phabricator.wikimedia.org/T292729 (10Tgr) @hashar The error started happening pretty consistently around 6AM UTC, so those patch...
[23:19:00] <wikibugs>	 (03CR) 10Ahmon Dancy: [C: 03+2] get_active_wikiversions: Sort wikiversions keys before scanning [tools/scap] - 10https://gerrit.wikimedia.org/r/739671 (owner: 10Ahmon Dancy)
[23:20:08] <wikibugs>	 (03Merged) 10jenkins-bot: get_active_wikiversions: Sort wikiversions keys before scanning [tools/scap] - 10https://gerrit.wikimedia.org/r/739671 (owner: 10Ahmon Dancy)
[23:21:21] <wmf-insecte>	 Yippee, build fixed!
[23:21:22] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #29651: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/29651/
[23:26:00] <wikibugs>	 10Phabricator, 10WMF-Legal: Phabricator Macros and other media are to have clear copyright status - https://phabricator.wikimedia.org/T128771 (10AntiCompositeNumber) | Macro | Apparent copyright status | |---|---| | [[https://phabricator.wikimedia.org/macro/view/46/|ambiguitytape]] | Unknown, likely unattribut...
[23:36:07] <wikibugs>	 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10Scap, 10Patch-For-Review: Improve efficiency of scap l10n operations - https://phabricator.wikimedia.org/T295304 (10dancy)
[23:37:27] <wikibugs>	 10Phabricator, 10WMF-Legal: Phabricator Macros and other media are to have clear copyright status - https://phabricator.wikimedia.org/T128771 (10Dzahn) cathug deleted by replacing it with https://commons.wikimedia.org/wiki/File:Key_delete.jpg (Public Domain!:)   https://phabricator.wikimedia.org/macro/view/44/
[23:49:46] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10ci-test-error (WMF-deployed Build Failure): TAR_ENTRY_ERROR ENOSPC: no space left on device - https://phabricator.wikimedia.org/T292729 (10Tgr) p:05Triage→03Unbreak!
[23:52:59] <tgr>	 ^ set to unbreak now as it's blocking all patches in GrowthExperiments at least; probably other repos with selenium tests as well
[23:59:55] <wikibugs>	 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for scherukuwada - https://phabricator.wikimedia.org/T293087 (10thcipriani) 05Open→03Resolved a:03ArielGlenn >>! In T293087#7498629, @SCherukuwada wrote: > Yes, I did attend one session where I got walked through a deplo...