[02:30:24] phag seems down with "Array to string conversion" on every page for me [02:30:27] phab* [02:30:55] fine for me [02:31:00] works for me [02:31:02] hm.. okay not every page [02:31:06] https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/defaults/43/ [02:31:10] I changed this form [02:31:14] now I can't open or edit it [02:31:20] https://phabricator.wikimedia.org/maniphest/task/edit/form/43/ [02:31:25] it's the bug report form [02:31:38] I just changed the description text a bit, nothing else.. [02:31:44] I guess some kind of roundtrip issue [02:31:48] ah yeah that's broken for me [02:31:55] better not edit any other forms in case it spreads and makes all task forms broken [02:32:25] Bug: Can not report bug, as there is a bug in the bug report form [02:41:20] Krinkle: https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/edit/43/ [02:41:32] can you revert there? [02:41:51] no, this subform only edits the title and description of the form, not the task template itself. [02:41:53] I copied the "edit form" link from another form and replaced the number [02:42:11] ack, the edit form link is https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/defaults/43/ [02:43:27] one is the edit form ("defaults"), the other is the meta form configuration ("edit") [02:43:39] https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/view/46/ [02:43:52] this is a similar task template, where you see the fields and task description template [02:44:27] which fails to view at https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/view/43/, fails to open at https://phabricator.wikimedia.org/maniphest/task/edit/form/43/ and fails to edit at https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/defaults/43/ [02:45:38] yea, I don't know how to revert it. maybe it just has to be replaced with a new form then. there is "Create Form" [02:47:13] people can still create tasks [02:51:19] yea, disable also doesnt work. only thing is reordering the forms in the menu [02:53:47] Krinkle: would "disable" be any better than broken? [02:54:01] https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/disable/43/ [02:54:37] I would suggest that for now.. then a task to create a replacement, I guess. [02:56:25] you're assuming new forms works and editing other forms won't break them the same way [02:56:40] I suspect it's a rountrip problem, so any submissino will end up saving sometihnb corrupt to that forms' configuration. [02:58:27] ok, ack [03:01:38] mutante: can you check the php error log on phab1? [03:01:59] the trace might help me figure out and prepare something for brennen or thcipriani when they're back [03:09:30] the /var/log/apache2/phabricator_error.log is full of stuff that looks unrelated to this form and moving fast [03:09:38] having a hard time to find it [03:09:55] that's where the PHP messages are though [03:12:40] ok, got something.. hold on [03:14:11] Krinkle: https://phabricator.wikimedia.org/P30961 [03:28:43] 10Phabricator (Upstream), 10Release-Engineering-Team: Task Form becomes inaccessible after edit - https://phabricator.wikimedia.org/T312614 (10Krinkle) [03:30:56] mutante: confirmed to break any form you edit [03:30:57] https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/view/89/ [03:31:04] this is the old "Create RFC" form that I archived a while back [03:31:10] save dummy :) [03:32:14] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream: Task Form becomes inaccessible after edit - https://phabricator.wikimedia.org/T312614 (10Krinkle) The last frame in the trace is `AphrontFormTextAreaControl.php` which has some [recent upstream changes](https://github.com/phacility/phabricator... [03:36:02] ugh, yes. ACK. just saw that ticket. really have to go now, been trying since hours to "just do one last thing" [03:36:51] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream: Task Form becomes inaccessible after edit - https://phabricator.wikimedia.org/T312614 (10Krinkle) Upstream issue that may be related: https://secure.phabricator.com/T13685 I suspect may be one of our custom form fields is configured incorrect... [07:14:04] 10Phabricator, 10Release-Engineering-Team: "RuntimeException: Array to string conversion" trying to use or edit form 17 - https://phabricator.wikimedia.org/T311956 (10Aklapper) [07:14:07] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream: Task Form becomes inaccessible after edit - https://phabricator.wikimedia.org/T312614 (10Aklapper) [07:15:22] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream: Task Form becomes inaccessible after edit - https://phabricator.wikimedia.org/T312614 (10Aklapper) p:05Triageβ†’03Unbreak! [07:29:57] 10Phabricator: [REQUEST] Disable one of my two Phabricator user accounts - https://phabricator.wikimedia.org/T193616 (10Aklapper) a:05CommunityTechBotβ†’03Aklapper [07:35:37] 10Phabricator: Archive my extra user account - https://phabricator.wikimedia.org/T312607 (10Aklapper) 05Openβ†’03Resolved I've disabled `@bgerdemann`. https://phabricator.wikimedia.org/p/bgerdemann/ is connected to a private self-created SUL account. https://phabricator.wikimedia.org/p/Bethany/ is connected to... [08:07:46] 10Scap: Can't sync vendor - https://phabricator.wikimedia.org/T301344 (10Legoktm) symfony has a tendency of doing this, they're not a fan of our "lint everything" strategy despite it stopping potential production issues. In the past we just deleted the offending file from vendor/ and added it to .gitignore until... [08:07:51] 10Scap, 10MediaWiki-Vendor: Can't sync vendor - https://phabricator.wikimedia.org/T301344 (10Legoktm) [08:53:40] Project beta-scap-sync-world build #58723: 04FAILURE in 8 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/58723/ [08:58:17] Project beta-scap-sync-world build #58724: 04STILL FAILING in 2 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/58724/ [09:06:24] Yippee, build fixed! [09:06:25] Project beta-scap-sync-world build #58725: 09FIXED in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/58725/ [09:07:53] 10Project-Admins: Create project tag for ML-Collab-2022 - https://phabricator.wikimedia.org/T312625 (10Slst2020) [09:40:25] 10Project-Admins: Create project tag for ML-Collab-2022 - https://phabricator.wikimedia.org/T312625 (10Aklapper) Hi, what is "ML"? Is this related to Machine Learning? I might go for `WMF-Inspiration-Week-2022-ML-Collab` then [09:58:48] 10Project-Admins: Create project tag for ML-Collab-2022 - https://phabricator.wikimedia.org/T312625 (10Slst2020) >>! In T312625#8064687, @Aklapper wrote: > Hi, what is "ML"? Is this related to Machine Learning? Yes, Machine Learning. > I might go for `WMF-Inspiration-Week-2022-ML-Collab` then Sounds good to me. [10:05:10] 10Phabricator, 10Release-Engineering-Team: Create Task -> "New Bug Report" results in an exception - https://phabricator.wikimedia.org/T312630 (10kostajh) [10:07:15] 10Phabricator, 10Release-Engineering-Team: Create Task -> "New Bug Report" results in an exception - https://phabricator.wikimedia.org/T312630 (10Aklapper) [10:07:24] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream: Task Form becomes inaccessible after edit - https://phabricator.wikimedia.org/T312614 (10Aklapper) [10:09:14] 10Project-Admins: Create project tag for ML-Collab-2022 - https://phabricator.wikimedia.org/T312625 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Requested public project #wmf-inspiration-week-2022-ml-collab has been created: https://phabricator.wikimedia.org/project/view/6023/ (In case you need to edit the... [10:12:06] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream: Task Form becomes inaccessible after edit - https://phabricator.wikimedia.org/T312614 (10Lucas_Werkmeister_WMDE) A possible workaround (suggested by @Dreamy_Jazz) is to create the task as a normal task and then change its subtype to Bug Report... [10:15:51] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream: Task Form becomes inaccessible after edit - https://phabricator.wikimedia.org/T312614 (10Dreamy_Jazz) I followed this for filing T312628 and have been able to interact with it since without errors. [11:55:40] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T308072 (10kostajh) @jnuche @hashar @thcipriani @Urbanecm_WMF there is a patch for review in {T312544} that should stop the errors/logsp... [12:06:07] urbanecm: thanks for the +2. hashar I think we should backport it today, if it's OK with RelEng. [12:06:36] ^ "it" = T312544 [12:06:36] T312544: Deferred update 'MWCallableUpdate_GrowthExperiments\NewcomerTasks\TaskSetListener->run' failed to run. - https://phabricator.wikimedia.org/T312544 [12:07:07] i agree. it's a lot of errors, and probably better to fix now. [12:27:32] ok, following the process from Deployments/Emergencies [13:19:02] kormat: urbanecm: sorry IRC was closed. Do you need any assistance for the deployment? [13:19:20] 10Release-Engineering-Team, 10CampaignEvents, 10Campaign-Registration, 10Campaign-Tools (Campaign-Tools-Sprint-16), 10Patch-For-Review: Release V0 of the CampaignEvents extension to betawiki - https://phabricator.wikimedia.org/T311752 (10ldelench_wmf) [13:19:34] hashar: i guess a releng (and SRE) approval :) [13:19:38] (cc kostajh) [13:19:58] happy to +1 or +2 && deploy whatever [13:20:16] lets synchronize in #wikimedia-operations [13:21:41] Hi all! Quick question on jenkins env for the tests, how can I tell from tox that I'm running on ci? (specifically on https://integration.wikimedia.org/ci/job/alerts-pipeline-test/460/console) [13:24:59] dcaro: Jenkins should set the `CI` environment variable in all jobs. Then tox has some custom code specific to jenkins (it looks up for a JENKINS_* env variable) and you can then have a custom override https://tox.wiki/en/latest/config.html#jenkins-override [13:25:23] `[tox:jenkins]` [13:25:32] which I think let you set a different env list or run a specific command [13:26:04] maybe envlist=py39 [13:27:33] dcaro: for scap I went to have blubber to define an image for each of Stretch, Buster, Bullseye and add the related pipelines https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/775400 [13:27:44] with pytest being invoked with whatever python 3 is shipped by the distro [13:28:12] and the CI config running three of them in parallel https://gerrit.wikimedia.org/r/c/integration/config/+/790747 [13:28:31] does that work also inside containers? (I see by the logs that the pipeline is actually running docker, but I don't see any ENV settings passed when building the image) [13:29:50] (sorry if I'm asking nonsens questions, my knowledge of the CI setup is limited xd) [13:30:43] hmm [13:30:48] good question :] [13:31:04] hmm, I can change the entrypoint though, and tell it "tox -e ci" or similar? [13:31:07] I don't know whether the env variable set by Jenkins are actually passed to the docker container spinned by Pipelinelib [13:31:25] and tox strips out most environment variable anyway [13:31:46] what are you trying to achieve? [13:32:57] I'm trying to run different tests (or with different options) locally and on ci [13:33:06] hashar urbanecm : I am afk but feel free to go forward with the patch deployment. I think it’s quite safe. I left instructions in the patch for verifying it. [13:33:12] Otherwise I can do it on Monday [13:33:16] kostajh: I might give it a try ;) [13:33:29] kostajh: given you have left info to reproduce, worse case I roll back [13:34:54] hashar: sounds good! feel free to ping if i can help. [13:37:57] dcaro: I guess you can try the `[tox:jenkins]` to have a different config set when tox detects `JENKINS_URL` being set. Assuming that env variable is passed to the container, that would let you alter the behavior when running on CI [13:38:40] I used that at https://gerrit.wikimedia.org/g/integration/quibble/+/refs/heads/master/tox.ini [13:38:48] [tox] [13:38:48] envlist = check, py3-lint, black, py3-unit, integration [13:39:00] let me run the test with whatever local python3 version I might have [13:39:13] but on CI it runs on the various python version sprovided by the image [13:39:13] [tox:jenkins] [13:39:13] envlist = check, py{35,36,37,38}-{lint}, black, py{35,36,37,38}-{unit}, doc [13:39:28] urbanecm: going to do the deploy and switching to -operations [13:39:36] ack [13:41:03] πŸ‘ will test! [13:49:52] 10GitLab (CI & Job Runners), 10serviceops, 10serviceops-collab, 10Patch-For-Review: DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Jelto) I'm seeing dns issues for jobs on Trusted Runners again: (example here https://gitlab.wikimedia.org/repos/releng/gitlab-tru... [13:50:20] (03PS1) 10Robert Vogel: Add new dependency to BlueSpicePageTemplates [integration/config] - 10https://gerrit.wikimedia.org/r/812323 [13:53:00] (03CR) 10CI reject: [V: 04-1] Add new dependency to BlueSpicePageTemplates [integration/config] - 10https://gerrit.wikimedia.org/r/812323 (owner: 10Robert Vogel) [14:14:44] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T308072 (10hashar) >>! In T308072#8065080, @kostajh wrote: > @jnuche @hashar @thcipriani @Urbanecm_WMF there is a patch for review in {T... [14:15:58] thanks for the backport hashar <3 [14:17:42] g'morning [14:18:28] acknowledging T312614; i'm not sure how much use i'm going to be for the next little while [14:18:28] T312614: Task Form becomes inaccessible after edit - https://phabricator.wikimedia.org/T312614 [15:11:46] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream: Task Form becomes inaccessible after edit - https://phabricator.wikimedia.org/T312614 (10brennen) I think this should represent the correct state for a deploy, with [[https://secure.phabricator.com/rPf2a7db1b1a1c3867f4a7b780a830608565caf3a2|(s... [15:14:16] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream: Task Form becomes inaccessible after edit - https://phabricator.wikimedia.org/T312614 (10brennen) Note that this did not include a `bin/celerity` run. Given the lack of CSS/JS updates, I think that should be irrelevant. [15:36:11] above will need phab downtimed & a deploy at some point. i'm getting on the road for an airport soon, so while i may be connected i probably won't have the kind of signal that leaves me comfortable driving a deploy for a bit. [15:44:58] ^ cc: thcipriani, mutante. going afk for a few, will check back in. we could also leave this 'til monday, judgment call to be made there i'd say. [15:45:36] brennen: ack, thank you, save travels [15:46:29] Farewell! [16:32:34] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream: Task Form becomes inaccessible after edit - https://phabricator.wikimedia.org/T312614 (10thcipriani) >>! In T312614#8065653, @brennen wrote: > On phab1001, we are fetching the deploy repo from gerrit: > > ` > sudo -u phab-deploy git fetch ger... [16:59:25] 10MediaWiki-Releasing, 10MediaWiki-Vendor, 10MW-1.37-release, 10Patch-For-Review: Wrong version of guzzle shipped with 1.37.3 (breaks update.php) - https://phabricator.wikimedia.org/T312059 (10Reedy) >>! In T312059#8053776, @Legoktm wrote: > @Reedy could we get a maintenance release for this? Having update... [17:02:12] 10Phabricator: Archive my extra user account - https://phabricator.wikimedia.org/T312607 (10Bethany) Thank you @Aklapper Duly noted about the Phab tag. [17:18:31] (03CR) 10Ahmon Dancy: [C: 03+2] Many changes to support mwpresync [tools/train-dev] - 10https://gerrit.wikimedia.org/r/810384 (owner: 10Ahmon Dancy) [17:19:01] (03Merged) 10jenkins-bot: Many changes to support mwpresync [tools/train-dev] - 10https://gerrit.wikimedia.org/r/810384 (owner: 10Ahmon Dancy) [17:27:06] (03PS2) 10Ahmon Dancy: git.clone_or_update_repo: Set core.sharedRepository=group [tools/scap] - 10https://gerrit.wikimedia.org/r/812099 [17:31:45] (03CR) 10CI reject: [V: 04-1] git.clone_or_update_repo: Set core.sharedRepository=group [tools/scap] - 10https://gerrit.wikimedia.org/r/812099 (owner: 10Ahmon Dancy) [17:32:11] 10MediaWiki-Releasing, 10MediaWiki-Vendor, 10MW-1.37-release: Wrong version of guzzle shipped with 1.37.3 (breaks update.php) - https://phabricator.wikimedia.org/T312059 (10Reedy) 05Openβ†’03Resolved a:03Reedy [17:32:19] 10MediaWiki-Releasing, 10MediaWiki-Vendor, 10MW-1.37-release: Wrong version of guzzle shipped with 1.37.3 (breaks update.php) - https://phabricator.wikimedia.org/T312059 (10Reedy) a:05Reedyβ†’03Zabe [17:35:33] (03PS3) 10Ahmon Dancy: Additional change for group sharing [tools/scap] - 10https://gerrit.wikimedia.org/r/812099 [19:52:25] 10GitLab, 10Release-Engineering-Team, 10serviceops, 10serviceops-collab, 10User-dduvall: Changes to modules/gitlab_runner/templates/config-template.toml.erb have no effect on existing runners - https://phabricator.wikimedia.org/T311746 (10dduvall) p:05Triageβ†’03Medium a:03dduvall [20:04:40] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream: Task Form becomes inaccessible after edit - https://phabricator.wikimedia.org/T312614 (10thcipriani) 05Openβ†’03Resolved a:03brennen [21:54:30] (03CR) 10Jeena Huneidi: [C: 03+2] "LGTM" [tools/scap] - 10https://gerrit.wikimedia.org/r/812099 (owner: 10Ahmon Dancy) [21:58:53] (03Merged) 10jenkins-bot: Additional change for group sharing [tools/scap] - 10https://gerrit.wikimedia.org/r/812099 (owner: 10Ahmon Dancy)