[00:01:58] <wikibugs>	 10Release-Engineering-Team (Radar), 10Release, 10Train Deployments: Investigate McRouter GET request spike from wmf.15 - https://phabricator.wikimedia.org/T310532 (10thcipriani) ## 1.39.0-wmf.15 code stuff >>! In T310532#8004202, @Jdforrester-WMF wrote: > ~~Is this just the UBN-ness of the back-ported fixes...
[00:25:40] <wikibugs>	 10Release-Engineering-Team (Radar), 10Release, 10Train Deployments: Investigate McRouter GET request spike from wmf.15 - https://phabricator.wikimedia.org/T310532 (10tstarling) Meeting with Krinkle, Aaron, we think it's because a [[https://gerrit.wikimedia.org/r/c/mediawiki/core/+/703560|recent change to zh...
[00:35:12] <wikibugs>	 10Release-Engineering-Team (Radar), 10Release, 10Train Deployments: Investigate McRouter GET request spike from wmf.15 - https://phabricator.wikimedia.org/T310532 (10tstarling) I'm working on a patch.
[01:03:40] <wikibugs>	 10Release-Engineering-Team (Radar), 10Performance-Team, 10Patch-For-Review, 10Release, 10Train Deployments: Investigate McRouter GET request spike from wmf.15 - https://phabricator.wikimedia.org/T310532 (10Krinkle) Here goes. Let's start with the actual source of the alert.  <https://grafana.wikimedia.or...
[02:31:04] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments, 10User-brennen: 1.39.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T308069 (10tstarling)
[02:31:09] <wikibugs>	 10Release-Engineering-Team (Radar), 10Performance-Team, 10MW-1.39-notes (1.39.0-wmf.17; 2022-06-20), 10Patch-For-Review, and 2 others: Investigate McRouter GET request spike from wmf.15 - https://phabricator.wikimedia.org/T310532 (10tstarling) 05Open→03Resolved Seems fixed.  {F35241960}
[04:30:29] <wikibugs>	 10Release-Engineering-Team (Radar), 10Performance-Team, 10MW-1.39-notes (1.39.0-wmf.15; 2022-06-06), 10Patch-For-Review, and 2 others: Investigate McRouter GET request spike from wmf.15 - https://phabricator.wikimedia.org/T310532 (10brennen) Thanks much all.  @Krinkle, that writeup is a thing of beauty and...
[07:44:11] <wikibugs>	 (03CR) 10Jaime Nuche: "This change is ready for review." [tools/scap] - 10https://gerrit.wikimedia.org/r/803556 (https://phabricator.wikimedia.org/T303559) (owner: 10Jaime Nuche)
[07:49:46] <wikibugs>	 (03PS1) 10Jaime Nuche: install-world: use bootstrap script from scap source dir [tools/scap] - 10https://gerrit.wikimedia.org/r/805727 (https://phabricator.wikimedia.org/T303559)
[07:52:30] <wikibugs>	 (03CR) 10Jaime Nuche: "Sorry about skipping the usual" [tools/scap] - 10https://gerrit.wikimedia.org/r/803556 (https://phabricator.wikimedia.org/T303559) (owner: 10Jaime Nuche)
[07:54:39] <wikibugs>	 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Patch-For-Review: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10SLyngshede-WMF)
[07:54:41] <wikibugs>	 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Allow deployers to sudo -u mwpresync - https://phabricator.wikimedia.org/T310654 (10SLyngshede-WMF) 05Open→03Resolved p:05Triage→03High a:03SLyngshede-WMF
[07:55:30] <wikibugs>	 (03CR) 10Jaime Nuche: "Sorry for skipping the normal review process. I need install-world to be stable before I go on vacation" [tools/scap] - 10https://gerrit.wikimedia.org/r/805727 (https://phabricator.wikimedia.org/T303559) (owner: 10Jaime Nuche)
[07:55:37] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+2] install-world: use bootstrap script from scap source dir [tools/scap] - 10https://gerrit.wikimedia.org/r/805727 (https://phabricator.wikimedia.org/T303559) (owner: 10Jaime Nuche)
[08:00:21] <wikibugs>	 (03Merged) 10jenkins-bot: install-world: use bootstrap script from scap source dir [tools/scap] - 10https://gerrit.wikimedia.org/r/805727 (https://phabricator.wikimedia.org/T303559) (owner: 10Jaime Nuche)
[08:04:54] <wikibugs>	 (03PS1) 10Jaime Nuche: Release 4.9.3-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/805735
[08:12:34] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+2] Release 4.9.3-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/805735 (owner: 10Jaime Nuche)
[08:17:11] <wikibugs>	 (03Merged) 10jenkins-bot: Release 4.9.3-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/805735 (owner: 10Jaime Nuche)
[08:20:55] <wikibugs>	 (03CR) 10Jaime Nuche: startup: warn user if not running from a Python virtual environment (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/803556 (https://phabricator.wikimedia.org/T303559) (owner: 10Jaime Nuche)
[09:00:51] <wikibugs>	 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10MarcoAurelio) I'd like to be able to manage my own personal project again: #user-marcoaurelio however the project settings are locked to #acl_project-admins and few other groups. Unt...
[09:16:23] <hashar>	 pfff finally sorted out that scap.cli.Application() testing
[09:16:28] <hashar>	 took me a while :-\
[09:23:46] <wikibugs>	 (03CR) 10Hashar: scap prep: set ssh push url on all repositories (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/805373 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar)
[09:23:53] <wikibugs>	 (03PS3) 10Hashar: scap prep: set ssh push url on all repositories [tools/scap] - 10https://gerrit.wikimedia.org/r/805373 (https://phabricator.wikimedia.org/T306425)
[09:26:01] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+1] "LGTM" [tools/scap] - 10https://gerrit.wikimedia.org/r/805373 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar)
[09:26:19] <hashar>	 jnuche: you wont believe how long it took me to figure out https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/805373/3/tests/scap/plugins/test_prep.py :D
[09:26:24] <hashar>	 probably 4 hours bah
[09:27:31] <jnuche>	 hashar: that happens with tests, but you figured it out! :)
[09:28:13] <wikibugs>	 (03CR) 10Hashar: [C: 04-2] "I have to refactor the test along the lines of https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/805373/3/tests/scap/plugins/test_pr" [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar)
[09:28:43] <hashar>	 jnuche: it took me a while to understand the magic in cli.Application , its factory and the custom scap.Args
[09:29:02] <jnuche>	 hashar: the way that the cli object gets created is a bit convoluted, took me a good deal of time too to find out how it works
[09:29:12] <hashar>	 or also that  `parser.parse_args(args=None)`   ends up using `sys.argv` (rather than the explicitly given `None`)
[09:29:32] <jnuche>	 buff, that's an ugly pitfall
[09:29:36] <hashar>	 I might add a decorator for testing purpose which will yield a fully configured application
[09:29:59] <jnuche>	 that's a great idea 👍
[09:30:04] <hashar>	 something like  `@scap("prep", "1.39.0-wmf.42")`
[09:30:19] <hashar>	 which would more or less do all the initialization steps from `Application.run()`
[09:30:22] <hashar>	 which well 
[09:30:25] <hashar>	 convenient `sys.exit`
[09:30:45] <hashar>	 and default to `sys.argv`  which leads to some fun like scap telling me "pytest" is not a valid command ahha
[09:57:24] <wikibugs>	 (03PS2) 10Hashar: Support a different user for Gerrit ssh interactions [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425)
[10:01:58] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Support a different user for Gerrit ssh interactions [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar)
[10:35:14] <wikibugs>	 10Project-Admins: Please add new tag "Ganeti" - https://phabricator.wikimedia.org/T310683 (10joanna_borun)
[10:46:42] <wikibugs>	 (03CR) 10Aklapper: [C: 03+1] "Please merge" [integration/docroot] - 10https://gerrit.wikimedia.org/r/791111 (https://phabricator.wikimedia.org/T302809) (owner: 10Aklapper)
[10:47:44] <wmf-insecte>	 Project beta-scap-sync-world build #55422: 04FAILURE in 0.98 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/55422/
[10:55:54] <hnowlan>	 hello! I'm getting an exception when running `scap sync-file`, happy to file a bug but just wanted to check whether this was known or an error on my part https://phabricator.wikimedia.org/P29785
[10:56:25] <wmf-insecte>	 Yippee, build fixed!
[10:56:26] <wmf-insecte>	 Project beta-scap-sync-world build #55423: 09FIXED in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/55423/
[11:18:51] <wikibugs>	 (03CR) 10Jaime Nuche: Support a different user for Gerrit ssh interactions (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar)
[11:20:35] <TheresNoTime>	 hnowlan: looks like a bug, maybe introduced in https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/scap/+/5e5b0f90d635df74b179d752d15a6c74f49ac691%5E%21/scap/main.py ? Could be worth trying with `--pause-after-testserver-sync` set?
[11:21:24] <wikibugs>	 (03PS3) 10Hashar: Support a different user for Gerrit ssh interactions [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425)
[11:30:05] <wikibugs>	 (03CR) 10Hashar: Support a different user for Gerrit ssh interactions (035 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar)
[11:30:13] <wikibugs>	 (03PS4) 10Hashar: Support a different user for Gerrit ssh interactions [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425)
[11:31:15] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Support a different user for Gerrit ssh interactions [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar)
[11:38:16] <wikibugs>	 (03PS4) 10Hashar: scap prep: set ssh push url on all repositories [tools/scap] - 10https://gerrit.wikimedia.org/r/805373 (https://phabricator.wikimedia.org/T306425)
[11:38:18] <wikibugs>	 (03PS5) 10Hashar: Support a different user for Gerrit ssh interactions [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425)
[11:39:03] <wikibugs>	 (03CR) 10Hashar: "Rebase cause the child change https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/805373/ had a conflict with the tip of the branch." [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar)
[11:39:36] <wikibugs>	 (03CR) 10Hashar: "Rebased due to a small conflict with I78a84f95915eab3a79e7d651bb4ef46328e86bda" [tools/scap] - 10https://gerrit.wikimedia.org/r/805373 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar)
[11:39:42] <hashar>	 rebase rebase
[11:43:22] <hashar>	 taavi: hi, about granting contint-admins access, I talked about it last week and indeed we need to answer properly. I will check again tonight with my US colleagues
[11:51:59] <wikibugs>	 10Phabricator: Rename my phabricator account to "Baggaet". - https://phabricator.wikimedia.org/T310651 (10Aklapper) 05Open→03Resolved a:03Aklapper Done.
[11:53:16] <duesen>	 How do I make the next week show up on https://wikitech.wikimedia.org/wiki/Deployments?
[11:53:31] <duesen>	 I want to schedule a deployment for next Monday
[12:25:10] <wikibugs>	 (03PS4) 10Jaime Nuche: startup: warn user if not running from a Python virtual environment [tools/scap] - 10https://gerrit.wikimedia.org/r/803556 (https://phabricator.wikimedia.org/T303559)
[12:25:12] <wikibugs>	 (03PS1) 10Jaime Nuche: release readme: update to reflect installation via scap-over-scap [tools/scap] - 10https://gerrit.wikimedia.org/r/805797 (https://phabricator.wikimedia.org/T303559)
[12:29:28] <wikibugs>	 (03PS1) 10Jaime Nuche: install-world: update minimum allowed scap version [tools/scap] - 10https://gerrit.wikimedia.org/r/805800 (https://phabricator.wikimedia.org/T303559)
[12:45:03] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] startup: warn user if not running from a Python virtual environment [tools/scap] - 10https://gerrit.wikimedia.org/r/803556 (https://phabricator.wikimedia.org/T303559) (owner: 10Jaime Nuche)
[13:06:14] <wikibugs>	 (03PS2) 10Jaime Nuche: release readme: update to reflect installation via scap-over-scap [tools/scap] - 10https://gerrit.wikimedia.org/r/805797 (https://phabricator.wikimedia.org/T303559)
[13:06:16] <wikibugs>	 (03PS2) 10Jaime Nuche: install-world: update minimum allowed scap version [tools/scap] - 10https://gerrit.wikimedia.org/r/805800 (https://phabricator.wikimedia.org/T303559)
[13:14:41] <awight>	 hashar: I had a thought about deployments.  It would be really nice if I could somehow bump all of the patches going out during a backport window, maybe with a code review comment, and the submit-gate CI would run up through the last step but without actually merging.  This way it all happens in parallel.  Then, as I'm prepared to deploy each patch I CR+2 and it immediately submits because the 
[13:14:47] <awight>	 tests have already passed...
[13:28:55] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform: Upgrade event platform related VMs in deployment-prep to Debian bullsye (or buster) - https://phabricator.wikimedia.org/T304433 (10Ottomata) 05Open→03Resolved
[13:28:58] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Ottomata)
[13:29:04] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10Ottomata)
[13:29:06] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Stretch Deprecation): Cloud VPS "deployment-prep" project Stretch deprecation - https://phabricator.wikimedia.org/T306068 (10Ottomata)
[13:31:07] <wikibugs>	 10Deployments: Scap sync-file fails with error about pause_after_testserver_sync - https://phabricator.wikimedia.org/T310699 (10awight)
[13:31:23] <wikibugs>	 10Deployments: Scap sync-file fails with error about pause_after_testserver_sync - https://phabricator.wikimedia.org/T310699 (10awight) p:05Triage→03Unbreak!
[13:33:44] <wikibugs>	 10Deployments: Scap sync-file fails with error about pause_after_testserver_sync - https://phabricator.wikimedia.org/T310699 (10awight) Can we revert https://github.com/wikimedia/scap/commit/5e5b0f90d635df74b179d752d15a6c74f49ac691 for now?
[13:39:23] <wikibugs>	 10Deployments: Scap sync-file fails with error about pause_after_testserver_sync - https://phabricator.wikimedia.org/T310699 (10awight) I was able to (unsafely) work around the issue by using the --force flag: `scap sync-file --force ...`
[13:39:58] <hnowlan>	 TheresNoTime: doesn't seem to get accepted as an arg :( I'll file a bug now
[13:40:29] <awight>	 hnowlan: I missed some backscroll, but maybe we're looking at the same thing? T310699
[13:40:30] <stashbot>	 T310699: Scap sync-file fails with error about pause_after_testserver_sync - https://phabricator.wikimedia.org/T310699
[13:40:34] <wikibugs>	 (03PS1) 10Jaime Nuche: Revert "scap backport: deploy to mwdebug" [tools/scap] - 10https://gerrit.wikimedia.org/r/805817 (https://phabricator.wikimedia.org/T310699)
[13:40:50] <hnowlan>	 awight: ah, yep! 
[13:40:56] <hnowlan>	 thanks for that 
[13:41:02] <awight>	 :-D likewise, it was helpful to see your paste
[13:41:16] <awight>	 "am I normal" is a typical question during my work day
[13:42:47] <wikibugs>	 10Deployments, 10Patch-For-Review: Scap sync-file fails with error about pause_after_testserver_sync - https://phabricator.wikimedia.org/T310699 (10jnuche) hi @awight Sorry about the issue. I'm going to revert the change
[13:43:23] <awight>	 jnuche: Thanks for dealing with this!  I'm unblocked for the immediate future, good luck.
[13:44:40] <jnuche>	 awight: glad that you found a quick workaround :)
[13:46:33] <awight>	 I don't see what the bug might be from looking at the code, but I was also unable to pass the --pause-after-testserver-sync flag.  Curious!
[13:48:22] <jnuche>	 yeah, by looking at the code, that value should always be populated, it's weird
[13:48:39] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+2] Revert "scap backport: deploy to mwdebug" [tools/scap] - 10https://gerrit.wikimedia.org/r/805817 (https://phabricator.wikimedia.org/T310699) (owner: 10Jaime Nuche)
[13:54:28] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "scap backport: deploy to mwdebug" [tools/scap] - 10https://gerrit.wikimedia.org/r/805817 (https://phabricator.wikimedia.org/T310699) (owner: 10Jaime Nuche)
[13:56:18] <wikibugs>	 (03PS1) 10Jaime Nuche: Release 4.9.4-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/805821
[14:02:25] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+2] Release 4.9.4-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/805821 (owner: 10Jaime Nuche)
[14:07:00] <wikibugs>	 (03Merged) 10jenkins-bot: Release 4.9.4-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/805821 (owner: 10Jaime Nuche)
[14:28:36] <hnowlan>	 thanks for the quick turnaround on fixing that issue!
[15:15:08] <icinga-wm>	 PROBLEM - https://phabricator.wikimedia.org #page on phabricator.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 5255 bytes in 0.024 second response time https://wikitech.wikimedia.org/wiki/Phabricator
[15:16:54] <icinga-wm>	 RECOVERY - https://phabricator.wikimedia.org #page on phabricator.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 39622 bytes in 0.146 second response time https://wikitech.wikimedia.org/wiki/Phabricator
[15:17:02] <hashar>	 ^^ known, it is being upgraded
[15:26:18] <dancy>	 !log Upgrading scap to 4.9.4-1+0~20220615151557.315~1.gbped3b8d in beta cluster
[15:26:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:30:30] <elukey>	 hey folks, I am bootstrapping a Cassandra cluster and getting
[15:30:31] <elukey>	 Error: /Stage[main]/Cassandra::Twcs/Scap::Target[cassandra/twcs]/Package[cassandra/twcs]: Provider scap3 is not functional on this host
[15:30:45] <elukey>	 is it something know? Before I review tons of puppet code :D
[15:31:04] <elukey>	 (not the specific repo but the Provider missing, never seen the error)
[15:39:08] <jnuche>	 elukey: your hosts should be bootstrapped by Puppet with a scap installation once they get added to the list of scap targets
[15:39:42] <jnuche>	 seems like there is some piece of configuration somewhere that is checking for the presence of scap, we probably need to remove that
[15:41:46] <elukey>	 jnuche: hi! When you mention `scap targets` you mean on the repo's scap config or elsewhere?
[15:41:49] <elukey>	 (just to be sure)
[15:42:40] <jnuche>	 the list at `/etc/dsh/group/scap_targets` on the deployment server
[15:43:30] <jnuche>	 my understanding is that list gets populated by Puppet from values in an etcd configuration somewhere, but I'm not sure exactly how a target is added
[15:46:23] <dancy>	 I guess the impression that /etc/dsh/group/scap_targets will list any host that uses the scap::target or mediawiki::scap class in puppet.
[15:46:29] <dancy>	 *I get the impression...
[15:47:01] <taavi>	 so sounds like the fix for elukey's problem is to run puppet on deploy1002?
[15:47:31] <elukey>	 taavi: trying, good point
[15:47:41] <jnuche>	 dancy: sry, yes you're right
[15:48:06] <taavi>	 dancy: is there a way for freshly imaged hosts to pull scap by themselves instead of waiting for deploy1002 to push to them?
[15:48:12] <jnuche>	 elukey: is your host ml-cache1001.eqiad.wmnet?
[15:48:20] <elukey>	 jnuche: it is yes
[15:48:59] <jnuche>	 ok, I don't see the bootstrapping in the logs, seems like it didn't trigger...
[15:49:05] <elukey>	 ah I see
[15:49:06] <elukey>	 /Stage[main]/Scap::Dsh/Scap::Dsh::Group[scap_targets]/File[/etc/dsh/group/scap_targets]/content) +ml-cache1001.eqiad.wmnet
[15:49:17] <jnuche>	 elukey: give me a sec, I'll fix it
[15:51:24] <jnuche>	 elukey: please try to run Puppet on your host again, now it should work
[15:52:24] <elukey>	 jnuche: yep it worked! What is the magic? (I have other hosts to run puppet on :D)
[15:52:27] <elukey>	 thanks a lot
[15:53:36] <jnuche>	 I had to run the boostrapping manually from deploy1002: `jnuche@deploy1002:~$ scap install-world --batch`
[15:54:03] <jnuche>	 but that should have run automatically, also I'm not sure what is causing Puppet to check for the package
[15:55:11] <elukey>	 ahh okok
[15:55:22] <jnuche>	 elukey: sorry about that, please let me know if you need help with the other hosts and we'll try to get the bootstrapping fixed
[15:55:54] <elukey>	 jnuche: nono please thanks a lot for the help, it would have taken me ages to figure it out.. tomorrow I'll try more hosts, in case is the command above ok to run anytime?
[15:55:58] <elukey>	 otherwise I'll ping you :)
[15:56:40] <jnuche>	 you can run it if you are in the `deployment` group
[15:57:00] <elukey>	 yep yep
[15:57:02] <elukey>	 super 
[15:57:04] <elukey>	 thanks again!
[15:57:14] <jnuche>	 I'm off tomorrow, but I'll keep an eye on IRC in case you need help, otherwise someone else from RelEng should be able to help
[15:57:18] <jnuche>	 no prob :)
[16:25:48] <wmf-insecte>	 maintenance-disconnect-full-disks build 395199 integration-agent-docker-1035 (/: 30%, /srv: 21%, /var/lib/docker: 97%): OFFLINE due to disk space
[16:28:33] <hashar>	 <none>                                                      <none>     9d50ff997e7b   About a minute ago   5.01GB
[16:28:33] <hashar>	 <none>                                                      <none>     1a05bb5dac6d   About a minute ago   5.01GB
[16:30:32] <hashar>	 something about ./gradlew ":datahub-frontend:dist"
[16:30:44] <wmf-insecte>	 maintenance-disconnect-full-disks build 395200 integration-agent-docker-1035 (/: 30%, /srv: 14%, /var/lib/docker: 62%): RECOVERY disk space OK
[16:31:35] <hashar>	 !log integration-agent-docker-1035: docker image prune
[16:31:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:32:05] <hashar>	 10G reclaimed
[16:55:38] <hashar>	 oh
[16:55:43] <hashar>	 wikibugs is probably broken somehow
[16:59:36] <hashar>	 dancy: I think the scap patches I wrote are ready for a final review/approval :] 
[16:59:46] <hashar>	 the parent change is https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/805373/
[17:04:32] <hashar>	 damn I need to rebase them all
[17:06:41] <hashar>	 better
[17:10:07] * Krinkle sees a new "Referenced Files" feature in the sidebar and likes it, e.g. at v
[17:10:08] * Krinkle sees a new "Referenced Files" feature in the sidebar and likes it, e.g. at https://phabricator.wikimedia.org/T308952
[17:14:00] <RhinosF1>	 Krinkle: it's from the security upgrades
[18:35:23] <brennen>	 handy
[19:11:16] <kostajh>	 hi, when I try to add a parent task (T222310) to https://phabricator.wikimedia.org/T310665 I get an error: "You cannot add more than 0 objects to the relationship."
[19:11:16] <stashbot>	 T222310: Homepage: impact module improvements - https://phabricator.wikimedia.org/T222310
[19:12:32] <kostajh>	 ^ I suppose that's related to the phabricator upgrade, which is why I'm commenting here :)
[19:15:16] <jynus>	 I can confirm- it also works the other way around (editing subtasks)
[19:15:24] <jynus>	 *fails
[19:15:42] <wikibugs>	 10Release-Engineering-Team, 10Scap: scap-o-scap: Bootstrapping a new host fails - https://phabricator.wikimedia.org/T310740 (10dancy)
[19:16:18] <dancy>	 gah!
[19:16:22] <dancy>	 damn upgrades
[19:17:03] <dancy>	 I'll file a bug for that.
[19:19:45] <taavi>	 dancy: looks like a custom hack of ours broken by some upstream change: https://github.com/wikimedia/phabricator/commit/4f4cae8e183d76bc758b5aaa5c966d5a607e027a
[19:21:11] <wikibugs>	 10Phabricator, 10Release-Engineering-Team: Editing tasks results in "You cannot add more than 0 objects to the relationship" error - https://phabricator.wikimedia.org/T310742 (10dancy)
[19:40:30] <hashar>	 dancy: if I get it right by moving "scap prep" to use `config["gerrit_url"]`  that might lets us provision train-dev with `scap prep master` ? 
[19:41:03] <dancy>	 `scap prep master` already worked in train-dev.
[19:41:22] <dancy>	 due to the existence of a ~/.gitconfig file w/ the insteadOf stuff
[19:41:22] <hashar>	 even for sending the patch back to gerrit?
[19:41:25] <hashar>	 AH
[19:41:49] <hashar>	 yeah so the same trick I found out ages ago and that we need to do in our ~/.gitconfig in prod
[19:41:57] <hashar>	 which the series of patches will render obsolete
[19:42:05] <dancy>	 however, the insteadOf stuff doesn't apply to  `git remote set-url` 
[19:42:14] <dancy>	 nod.
[19:42:22] <dancy>	 Yeah, I like the changes you're making.
[19:44:03] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Nice and kudos on releasing the developer portal :)" [integration/docroot] - 10https://gerrit.wikimedia.org/r/791111 (https://phabricator.wikimedia.org/T302809) (owner: 10Aklapper)
[19:45:12] <wikibugs>	 (03Merged) 10jenkins-bot: doc: Add Developer Portal [integration/docroot] - 10https://gerrit.wikimedia.org/r/791111 (https://phabricator.wikimedia.org/T302809) (owner: 10Aklapper)
[19:45:32] <wikibugs>	 10Phabricator, 10Release-Engineering-Team: Editing tasks results in "You cannot add more than 0 objects to the relationship" error - https://phabricator.wikimedia.org/T310742 (10Dylsss) It wasn't broken, it was only deployed today and the change doesn't work properly because only `ManiphestTaskMergeInRelations...
[19:48:58] <wikibugs>	 10Phabricator, 10Release-Engineering-Team, 10Regression: Editing tasks results in "You cannot add more than 0 objects to the relationship" error - https://phabricator.wikimedia.org/T310742 (10Aklapper) p:05Triage→03High
[19:51:19] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Deployed with scap.  The web cache frontend has a one hour TTL so it might take a bit of time for the change to reflect on https://doc.wik" [integration/docroot] - 10https://gerrit.wikimedia.org/r/791111 (https://phabricator.wikimedia.org/T302809) (owner: 10Aklapper)
[19:54:20] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Update Wikibase section (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/802138 (owner: 10Lucas Werkmeister (WMDE))
[20:04:10] <wikibugs>	 10Phabricator, 10Release-Engineering-Team, 10Regression: Editing tasks results in "You cannot add more than 0 objects to the relationship" error - https://phabricator.wikimedia.org/T310742 (10brennen) Ok, think I have a patch:  `lines=10 --git a/src/applications/maniphest/relationship/ManiphestTaskMergeInRel...
[20:08:47] <RhinosF1>	 brennen: your patch files need attaching to the task
[20:08:53] <RhinosF1>	 They are private by default now
[20:09:44] <wikibugs>	 (03CR) 10Aklapper: "Thank you hashar!" [integration/docroot] - 10https://gerrit.wikimedia.org/r/791111 (https://phabricator.wikimedia.org/T302809) (owner: 10Aklapper)
[20:14:05] <wikibugs>	 10Phabricator, 10Release-Engineering-Team, 10Regression: Editing tasks results in "You cannot add more than 0 objects to the relationship" error - https://phabricator.wikimedia.org/T310742 (10brennen)
[20:14:11] <wikibugs>	 10Phabricator, 10Release-Engineering-Team, 10Regression: Editing tasks results in "You cannot add more than 0 objects to the relationship" error - https://phabricator.wikimedia.org/T310742 (10brennen)
[20:14:17] <brennen>	 RhinosF1: ah, interesting
[20:27:35] <RhinosF1>	 brennen: id say confusing
[20:28:56] <Reedy>	 It'd be useful to have a "make public" type button on the File pages
[20:30:44] <brennen>	 hrm, i accidentally included a change on https://phabricator.wikimedia.org/D1202 that wasn't in the original git commit.  how do i amend this?
[20:31:50] <brennen>	 ah, think i got it.
[20:31:55] <Reedy>	 abandon and start again
[20:32:20] <brennen>	 seems like if you amend the git commit and run `arc diff` again you're good.
[20:45:13] <RhinosF1>	 Reedy: there probably is somewhere
[20:45:22] <RhinosF1>	 I think there's something in attachments?
[20:45:39] <RhinosF1>	 I've seen how to unattach but not attach a file to a task
[20:46:08] <Reedy>	 If you go to the file, edit, change visible to...
[20:47:32] <RhinosF1>	 Reedy: is there not anything in the file attachments too?
[20:48:17] <brennen>	 i appreciate that "Foist Upon" is an action i can take with a revision in phabricator.
[20:48:34] <RhinosF1>	 I like phab's humoury stuff
[20:57:42] <wikibugs>	 10Release-Engineering-Team, 10Scap: scap-o-scap: Bootstrapping a new host fails - https://phabricator.wikimedia.org/T310740 (10dancy)
[21:14:18] <Dylsss>	 RhinosF1: If the file view policy is more restrictive then the view policy of an object that a file is referenced in (and you can view the file), then there should be a button "File Not Attached" on the object that you can click to attach.
[21:15:33] <Dylsss>	 https://usercontent.irccloud-cdn.com/file/8xd1grH4/image.png
[21:23:14] <brennen>	 yeah, that's what i just used on those screenshots i uploaded.
[21:23:35] <brennen>	 bit of an odd workflow considering i uploaded them while composing a comment on that task.
[21:25:01] <RhinosF1>	 Dylsss: yeah, the problem that's been seen is that the default upload policy is only uploader
[21:25:14] <RhinosF1>	 Which is kinda unexpected
[21:25:20] <RhinosF1>	 And confused people
[21:53:52] <wikibugs>	 (03PS1) 10Ahmon Dancy: scap backport: deploy to mwdebug (revised) [tools/scap] - 10https://gerrit.wikimedia.org/r/805763 (https://phabricator.wikimedia.org/T308476)
[22:02:36] <brennen>	 mutante: re: T310742, wondering how disruptive it will be to deploy this
[22:02:37] <stashbot>	 T310742: Editing tasks results in "You cannot add more than 0 objects to the relationship" error - https://phabricator.wikimedia.org/T310742
[22:03:08] <brennen>	 it seems like it requires a full deploy, although it should be much briefer than one involving migrations.
[22:04:33] <brennen>	 the main thing i don't know is how to avoid a page.
[22:24:25] <wikibugs>	 10Release-Engineering-Team, 10Wikimedia-GitHub: Add Matthew Vernon (@mcv21) to Wikimedia github - https://phabricator.wikimedia.org/T288350 (10MarcoAurelio)
[22:24:39] <wikibugs>	 (03PS2) 10Ahmon Dancy: scap backport: deploy to mwdebug (revised) [tools/scap] - 10https://gerrit.wikimedia.org/r/805763 (https://phabricator.wikimedia.org/T308476)
[22:26:54] <wikibugs>	 (03CR) 10Ahmon Dancy: "This is the same as https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/803370 but it handles the case where subclasses of AbstractSyn" [tools/scap] - 10https://gerrit.wikimedia.org/r/805763 (https://phabricator.wikimedia.org/T308476) (owner: 10Ahmon Dancy)
[22:39:50] <brennen>	 !log phabricator: tagged release/2022-06-15/1 (T310742)
[22:39:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[22:39:52] <stashbot>	 T310742: Editing tasks results in "You cannot add more than 0 objects to the relationship" error - https://phabricator.wikimedia.org/T310742
[22:55:01] <TheresNoTime>	 whew I thought I was losing it for a sec then.. ref T310742
[22:55:02] <stashbot>	 T310742: Editing tasks results in "You cannot add more than 0 objects to the relationship" error - https://phabricator.wikimedia.org/T310742
[22:56:10] <TheresNoTime>	 if y'all want a data point, I can confirm trying it like 10 times in a row doesn't fix it </sarcasm> :')
[23:01:34] <brennen>	 :D
[23:23:20] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.39.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T308069 (10brennen) Noting {T310742} here - if you need to add a blocker task before we get that fix out, please mark...