[00:07:42] 10Release-Engineering-Team, 10MW-on-K8s, 10SRE, 10serviceops: Check out www-portals repo in the mediawiki-webserver and in the mediawiki-multiversion images - https://phabricator.wikimedia.org/T285325 (10jeena) a:03jeena [06:01:17] 10Release-Engineering-Team, 10MW-on-K8s: The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Joe) >>! In T285232#7170223, @dduvall wrote: > Which image are you looking at? I'm looking at a recent image published as `docker-registry.discover... [06:36:17] (03CR) 10Lars Wirzenius: [C: 03+1] "I'm not able to test this, so I won't +2 it, but looks good to me." [tools/scap] - 10https://gerrit.wikimedia.org/r/700520 (https://phabricator.wikimedia.org/T192703) (owner: 10Hashar) [07:15:36] awight: quibble question if you have a moment -- what value do you use for ZUUL_URL? I keep getting timeouts when trying to clone repos [07:19:59] I eventually get `https://gerrit.wikimedia.org/r/mediawiki/core/': Failed to connect to gerrit.wikimedia.org port 443: Connection timed out` [07:23:50] (03PS1) 10Kosta Harlan: zuul: Add quibble-apache-selenium job to experimental group for GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/701050 (https://phabricator.wikimedia.org/T276428) [07:24:58] (03PS2) 10Kosta Harlan: zuul: Add Apache/Selenium job to experimental group for GrowthExperiments and VE [integration/config] - 10https://gerrit.wikimedia.org/r/701050 (https://phabricator.wikimedia.org/T276428) [07:25:02] (03CR) 10jerkins-bot: [V: 04-1] zuul: Add Apache/Selenium job to experimental group for GrowthExperiments and VE [integration/config] - 10https://gerrit.wikimedia.org/r/701050 (https://phabricator.wikimedia.org/T276428) (owner: 10Kosta Harlan) [07:28:24] 10Phabricator: Disable Slowvote application - https://phabricator.wikimedia.org/T285304 (10Aklapper) Sure, it can be sometimes. :) See the points I made in my previous comment though. [07:47:47] 10Continuous-Integration-Infrastructure: mwgate-node12-docker consistently fails for EventLogging - https://phabricator.wikimedia.org/T285372 (10Gilles) [07:48:17] 10Continuous-Integration-Infrastructure: mwgate-node12-docker consistently fails for EventLogging - https://phabricator.wikimedia.org/T285372 (10Gilles) [07:48:33] awight: oh, well of course today it "just works" :) [08:11:12] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Aklapper) @XenoRyet: Added you. //Usual disclaimer: Please follow [guidelines](https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects#Creating_new_projects) (namin... [08:31:16] kostajh: I somehow don't have to specify ZUUL_URL. fwiw, I have a quibble workspace/ref directory which I manually update once in a while. Not ideal for pulling new patches though [08:36:33] awight: hmm, if I omit ZUUL_URL from the .env file, quibble complains that it's required [08:45:07] * awight scratches head and does not find an .env file [08:47:38] awight: oh, how are you reproducing CI builds? [08:47:57] I'm getting the timeout errors again, it looks like the failing part is "git remote prune --dry-run origin". I don't know why that is even running [08:48:22] https://doc.wikimedia.org/quibble/#reproducing-a-ci-build says to use a .env file [08:48:43] yeah sorry I rarely do the entire CI reproduction thing, that's the difference [08:49:40] but if you check out the right versions locally, I think you'll be able to get pretty good results with --skip-zuul [08:50:30] +1 tickling zuul is an area that should be automated further. [08:50:39] (03CR) 10Hashar: "There is a similar pattern for phpunit:unit, consider using the same one using the new phpunit:entrypoint ?" (033 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/692372 (https://phabricator.wikimedia.org/T90875) (owner: 10Kosta Harlan) [08:50:55] I miss the backlog [08:51:18] kostajh: yeah the reproducing a CI build is broken I think [08:51:46] cause the Zuul system merges patches against the tip of branches in git repositories which are on contint1001 / contint2001 but are not publicly accessible [08:52:02] right [08:52:09] I guess it is all about adding a public git-daemon (possibly behind apache, varnish etc) to expose the private repo [08:52:18] well, so I "just" want to run a patch for GrowthExperiments with quibble locally to see why it's failing. What should I do? [08:52:36] and there is the list of dependencies to be injected which is really a mess [08:52:58] what I do is that I look at the build, I copy the EXT_DEPENDENCIES / SKIN_DEPENDENCIES env variables to my local invocation [08:53:20] and I point ZUUL_URL=https://gerrit.wikimedia.org/r/ [08:53:31] ZUUL_REF=refs/changes/45/12345/99 [08:53:39] aka fetch the patch from Gerrit directly [08:54:03] it slightly different from what Zuul has crafted since it first merge the patches against the tip of the branch [08:54:25] AND does set internal references for any other repositories that had a patch ahead or mentioned via Depends-On [08:54:34] but for a single standalone patch, that usually does it [08:55:14] that's the same as what I'm doing, excpet I'm pointing to a .env file. I guess you're just saying that you do `--skip-zuul`? [09:00:51] sigh, now I get connection refused when trying to pull a newer image from wikimedia docker registry. but that's probably some docker for mac issue [09:02:11] kostajh: --skip-zuul would skip cloning the repositories / resetting repo, fetching the patch [09:02:26] so typically I would invoke quibble to setup the repositories and dependencies (vendor or composer) [09:02:47] and on the next invocations pass --skip-zuul --skip-deps [09:02:56] it is definitely not a good UX [09:03:36] ok... now I'm seeing a weird issue in both docker container or when I try to run on my host. probably missing something obviou [09:04:03] I have ~/quibble which contains ref, src, log, etc directories. src is mediawiki core [09:04:58] if I go into ~/quibble/src and run `quibble --skip-zuul` I get `No such file or directory: '/Users/kostajh/quibble/src/src/composer.json'` [09:05:09] if I go to ~/quibble and run the same command, I get `No composer.json in current directory, do you want to use the one at /Users/kostajh/quibble/src? [Y,n]?` [09:05:47] eeeeeeek [09:06:20] locally I run quibble in a standalone directory: ~/workspace [09:06:45] which has cache, log, src directories [09:07:25] with all gerrit repositories under ~/projects I thus: [09:07:32] cd ~/workspace; quibble --git-cache ~/projects/ [09:08:37] --workspace represents the build directory on Jenkins, and I think the git clones are done under /src/ relatively to that workspace option [09:08:53] which default to the current directory [09:09:01] right [09:09:07] so yeah hmm in your case, you should run quibble from ~/quibble [09:09:28] so I'm basically doing the same thing, but I am getting the prompt about composer.json not being in the current directory [09:09:41] as to why it cant find ~/quibble/src/composer.json , I cant tell. Maybe the dir is empty? [09:09:51] or there is some path issue somewhere [09:09:59] no, it has a checkout of mediawiki core [09:10:55] or it runs a composer command from the current working directory ( ~/quibble ) [09:11:10] instead of from self.mw_install_path ( = ~/quibble/src on your setup) [09:11:40] yeah. I changed the directory name to workspace in case there was somehting silly there. [09:15:54] so why is it running composer commands from ~/workspace instead of from ~/workspace/src ? [09:27:14] kostajh: no idea :-\ [09:28:27] If i run `quibble --git-parallel=8 --packages-source=composer --git-cache ~/workspace/ref --db=sqlite --run=selenium` it doesn't complain about wrong working directory [09:28:40] so I guess it's something about the --packages-source=vendor bit? [09:29:12] hmm [09:30:54] maybe that fails when doing the step "Install composer dev-requires for vendor.git" [09:31:06] if for example ~/workspace/src/vendor/ had no composer.json [09:31:17] hmm no [09:31:44] ah yeah [09:31:59] that does run from the vendor directory [09:40:53] 10Release-Engineering-Team (Next), 10ContentSecurityPolicy, 10GitLab (Initialization), 10User-brennen: Define a Content Security Policy for GitLab - https://phabricator.wikimedia.org/T285363 (10hashar) [09:41:30] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap: Localisation cache is broken in beta - https://phabricator.wikimedia.org/T285345 (10hashar) [09:42:31] hashar: iirc you have a trick to have quibble pause its execution once it starts the server, is that right? [09:44:00] kostajh: quibble -c bash [09:44:07] drops you in an interactive shell [09:44:09] cool [09:44:16] with tentatively all backend services spawned [09:44:32] and I think I got some MW* env variables set as well so one can easily run wdio from the cli [09:44:42] should probably add that to the doc ;D [09:46:11] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap: Localisation cache is broken in beta - https://phabricator.wikimedia.org/T285345 (10hashar) Looks like most of the investigation so far is T285125#7163052 and following comments. [09:53:25] off for lunch etc [10:18:46] (03PS3) 10Kosta Harlan: zuul: Add Apache/Selenium job to experimental group for GrowthExperiments and VE [integration/config] - 10https://gerrit.wikimedia.org/r/701050 (https://phabricator.wikimedia.org/T276428) [11:29:29] (03PS1) 10Jbond: operations-puppet: Refresh to update gems [integration/config] - 10https://gerrit.wikimedia.org/r/701079 [11:31:47] (03CR) 10Jbond: "The puppet change has allready been merged so this can go anytime" [integration/config] - 10https://gerrit.wikimedia.org/r/701079 (owner: 10Jbond) [12:11:04] (03CR) 10Jdrewniak: [C: 03+1] Switch portals/deploy to NodeJS 12 [integration/config] - 10https://gerrit.wikimedia.org/r/700911 (https://phabricator.wikimedia.org/T285239) (owner: 10Hashar) [12:19:18] (03CR) 10Daimona Eaytoy: phpunit: Add support for vendor/bin/phpunit entrypoint (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/692372 (https://phabricator.wikimedia.org/T90875) (owner: 10Kosta Harlan) [12:30:58] (03CR) 10Kosta Harlan: phpunit: Add support for vendor/bin/phpunit entrypoint (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/692372 (https://phabricator.wikimedia.org/T90875) (owner: 10Kosta Harlan) [12:31:43] (03CR) 10Daimona Eaytoy: phpunit: Add support for vendor/bin/phpunit entrypoint (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/692372 (https://phabricator.wikimedia.org/T90875) (owner: 10Kosta Harlan) [12:50:15] 10Continuous-Integration-Infrastructure: mwgate-node12-docker consistently fails for EventLogging - https://phabricator.wikimedia.org/T285372 (10Ottomata) Makes sense, we recently switched eventgate CI to using service-pipeline-test, because node12 and the version of node-rdkafka + librdkafka we use are not yet... [14:24:58] 10Continuous-Integration-Infrastructure: mwgate-node12-docker consistently fails for EventLogging - https://phabricator.wikimedia.org/T285372 (10Krinkle) Why would CI for this mediawiki extension need to install a kafka client? I thought this was just a side-effect of the package.json mixup. [14:28:12] 10Continuous-Integration-Infrastructure: mwgate-node12-docker consistently fails for EventLogging - https://phabricator.wikimedia.org/T285372 (10Ottomata) Hm, Ok I'm just guessing here, but maybe because `@wikimedia/node-rdkafka-factory` is an optionalDependency, and the CI doesn't run `npm install --no-optiona... [14:56:27] (03PS1) 10David Caro: tox-poetry-buster: Added a new image for poetry based builds [integration/config] - 10https://gerrit.wikimedia.org/r/701121 [14:57:27] (03CR) 10Hashar: "> Unfortunately gitiles.jar is already bundled in gerrit.war and would thus be overwritten when updating Gerrit :-\" [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/700932 (https://phabricator.wikimedia.org/T240264) (owner: 10Hashar) [14:57:56] (03CR) 10jerkins-bot: [V: 04-1] tox-poetry-buster: Added a new image for poetry based builds [integration/config] - 10https://gerrit.wikimedia.org/r/701121 (owner: 10David Caro) [15:18:07] (03PS2) 10David Caro: tox-poetry-buster: Added a new image for poetry based builds [integration/config] - 10https://gerrit.wikimedia.org/r/701121 [15:47:31] 10Gerrit, 10Release-Engineering-Team (Doing): Upgrade Gerrit to 3.2.10 - https://phabricator.wikimedia.org/T278990 (10hashar) Last week was Wikimedia all-hands and thus had little activity. I have reached out to Gerrit upstream, that bug is definitely a regression and they are willing to release it quickly, t... [15:51:09] 10LibUp, 10MediaWiki-Releasing, 10Release-Engineering-Team (Seen), 10MediaWiki-Release-Tools: Work out why initial skin/extension branches into REL1_35/REL1_36 were on the wrong branch - https://phabricator.wikimedia.org/T283883 (10Reedy) >>! In T283883#7122591, @Reedy wrote: >>>! In T283883#7122580, @Umhe... [16:21:47] 10MediaWiki-Releasing, 10Security: Write and send release announcements for MediaWiki 1.32.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279728 (10Reedy) 05Open→03Resolved [16:21:51] 10MediaWiki-Releasing, 10Security: Write and send release announcements for MediaWiki 1.32.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279728 (10Reedy) [16:32:10] !log beta update jobs have been stuck for ~9.5 hours. Going to attempt to unstick [16:32:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:46:25] 10MediaWiki-Releasing, 10Security: Obtain CVEs for 1.32.15/1.35.3/1.36.1 security releases - https://phabricator.wikimedia.org/T279732 (10Reedy) [16:54:37] Reedy: git tags? [16:55:10] RhinosF1: Yes. If you're watching, CI has only really just merged the last pach [16:55:12] *patch [16:55:18] ok [16:55:20] In the last ~5 minutes [16:55:34] They're pushed now [16:56:19] * RhinosF1 groans github replication [16:56:30] seen [16:56:48] 10MediaWiki-Releasing, 10Security: Tracking bug for MediaWiki 1.32.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279726 (10Reedy) [16:56:54] 10MediaWiki-Releasing, 10Security: Tracking bug for MediaWiki 1.32.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279726 (10Reedy) 05Open→03Resolved a:03Reedy [16:59:38] 10MediaWiki-Releasing, 10Security: Update onwiki release notes for 1.32.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279729 (10Reedy) 05Open→03Resolved a:03Reedy [16:59:44] 10MediaWiki-Releasing, 10Security: Update onwiki release notes for 1.32.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279729 (10Reedy) [17:00:13] 10MediaWiki-Releasing, 10Security: Tag 1.32.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279731 (10Reedy) 05Open→03Resolved ` $ git push --tags Enumerating objects: 3, done. Counting objects: 100% (3/3), done. Delta compression using up to 2 threads Compressing objects: 100% (3/3), done. Writing o... [17:00:16] 10MediaWiki-Releasing, 10Security: Tag 1.32.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279731 (10Reedy) [17:00:44] 10MediaWiki-Releasing, 10Security: Update HISTORY in master after 1.32.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279730 (10Reedy) [17:01:21] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.32.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279725 (10Reedy) [17:06:01] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.32.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279725 (10Reedy) [17:06:07] 10MediaWiki-Releasing, 10Patch-For-Review, 10Security: Update HISTORY in master after 1.32.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279730 (10Reedy) 05Open→03Resolved a:03Reedy [17:08:11] and it all looks. good [17:11:11] (03PS1) 10Reedy: Update makesecuritytasks.py [tools/release] - 10https://gerrit.wikimedia.org/r/701160 [17:12:30] (03CR) 10jerkins-bot: [V: 04-1] Update makesecuritytasks.py [tools/release] - 10https://gerrit.wikimedia.org/r/701160 (owner: 10Reedy) [17:13:53] (03PS2) 10Reedy: Update makesecuritytasks.py [tools/release] - 10https://gerrit.wikimedia.org/r/701160 [17:18:29] (03CR) 10jerkins-bot: [V: 04-1] Update makesecuritytasks.py [tools/release] - 10https://gerrit.wikimedia.org/r/701160 (owner: 10Reedy) [17:19:38] (03PS3) 10Reedy: Update makesecuritytasks.py [tools/release] - 10https://gerrit.wikimedia.org/r/701160 [17:22:23] (03CR) 10Reedy: [C: 03+2] Update makesecuritytasks.py [tools/release] - 10https://gerrit.wikimedia.org/r/701160 (owner: 10Reedy) [17:23:46] (03Merged) 10jenkins-bot: Update makesecuritytasks.py [tools/release] - 10https://gerrit.wikimedia.org/r/701160 (owner: 10Reedy) [19:29:59] 10Project-Admins, 10MediaViewer, 10Structured-Data-Backlog, 10PM: Convert MediaViewer tracking tasks to Phab workboard columns; use board columns for categories only instead of progress tracking - https://phabricator.wikimedia.org/T282946 (10Aklapper) @CBogen: Any comments from #Structured-Data-Backlog team? [19:32:52] 10Project-Admins, 10AutoWikiBrowser: Convert AWB tracker tasks to Phabricator project tags - https://phabricator.wikimedia.org/T282945 (10Aklapper) @Reedy, @Rjwilmsi: Any comments? [19:34:00] 10Project-Admins, 10MediaViewer, 10Structured-Data-Backlog, 10PM: Convert MediaViewer tracking tasks to Phab workboard columns; use board columns for categories only instead of progress tracking - https://phabricator.wikimedia.org/T282946 (10CBogen) >>! In T282946#7173462, @Aklapper wrote: > @CBogen: Any c... [19:58:43] 10Release-Engineering-Team (Logspam), 10Platform Engineering, 10Wikimedia-production-error: PHP Notice: Undefined index: frameCount - https://phabricator.wikimedia.org/T285431 (10dduvall) p:05Triage→03Unbreak! [20:00:22] 10Release-Engineering-Team (Doing), 10GitLab (Initialization), 10User-brennen: Remove Speed & Function blockers for GitLab work - https://phabricator.wikimedia.org/T274458 (10brennen) [20:00:48] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10GitLab (Initialization), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10brennen) [20:01:58] 10Release-Engineering-Team (Logspam), 10Platform Engineering, 10Wikimedia-production-error: PHP Notice: Undefined index: frameCount - https://phabricator.wikimedia.org/T285431 (10dduvall) [20:02:00] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T281152 (10dduvall) [20:07:34] 10Release-Engineering-Team (Logspam), 10Platform Engineering, 10User-Ladsgroup, 10Wikimedia-production-error: PHP Notice: Undefined index: frameCount - https://phabricator.wikimedia.org/T285431 (10Ladsgroup) a:03Ladsgroup I'll handle it after eating dinner if no one beats me to it [20:32:02] 10Release-Engineering-Team (Logspam), 10Platform Engineering, 10User-Ladsgroup, 10Wikimedia-production-error: PHP Notice: Undefined index: frameCount - https://phabricator.wikimedia.org/T285431 (10Krinkle) It seems in PNGMetadataExtractor, `$frameCount` is always defined and declared in the metadata array,... [20:34:08] hi releng! Any ideas why this patch doesn't get picked up by zuul? https://gerrit.wikimedia.org/r/c/mediawiki/core/+/701168 [20:34:43] I can manually V+2 in that branch, but I'm pretty sure we had CI running for it as recently as yesterday [20:35:27] yah, jenkins-bot handled yesterday's fine: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/700973 [20:35:50] Would it be because the new one is a merge (from upstream branch REL1_35) ? [20:42:08] ejegg: might be :D [20:42:18] let me dig in zuul logs [20:44:40] ejegg: that new patchset hasn't reached zuul somehow :-\ [20:51:28] weird stuff [20:52:39] well if I send a second patch on top of your https://gerrit.wikimedia.org/r/c/mediawiki/core/+/701178 it does get recognized [20:53:17] huh. Well, we only merge from upstream when there's a security release every few months, so for now I'm happy to V+2 manually [20:53:24] thanks for checking that out [20:53:37] I guess it is something broken inside Gerrit internal somehow [20:53:45] * ejegg shrug [20:54:20] if I got to your change, under the list of files and change the parent diff to Auto Merge it fails [20:54:53] oh hmm, I just manually submitted in any case [20:55:03] sometime yeah :-\ [20:55:38] ah [20:55:47] Couldn't fire event [CONTEXT SUBMISSION_ID="701168-1624481636847-633ff185" project="mediawiki/core" ] [20:55:56] org.eclipse.jgit.errors.MissingObjectException: Missing blob 6a69f1ade607add20b119bd91fd34525edf1049c [20:56:09] so yeah jgit is not happy about your change cause it misses something [20:56:20] and the event is never dispatched to zuul/ci [20:56:25] and stuff is broken here and there [20:56:36] oh shoot, missing blob? [20:57:03] weird, the deployment mechanism checked it out (+submodules) just fine [20:57:29] let's see where that blob is supposed to be... [20:58:03] seems to be a CategoryTree extension ref [20:58:29] and I guess that didn't bother our deployment tools because we've deleted that submodule in our branch [20:59:42] maybe jgit get confused when a merge brings in a submodule deletion [20:59:49] could be [21:00:05] the follow up patch I have send on top of your change fails https://gerrit.wikimedia.org/r/c/mediawiki/core/+/701178/2 [21:00:39] 1) DonationInterface_Adapter_GlobalCollect_GlobalCollectTest::testNormalizeOrderID [21:00:43] 00:03:54.107 assertInternalType() is deprecated and will be removed in PHPUnit 9. Refactor your test to use assertIsNumeric() instead. [21:00:44] etc [21:01:02] guess cause mediawiki/core switched to some newer phpunit and some of your test need an update as a result [21:01:14] and [21:01:15] 3) DonationInterface_FraudFiltersTest::testMinFraudErrorScore [21:01:15] 00:03:54.107 Method post may not return value of type string, its return declaration is ": array" [21:01:23] oh dang, ok, we should fix those [21:01:46] which is what CI would have caught on your change if it actually had been able to receive the event [21:01:57] so I guess you can just create a change that fixes thoses [21:02:03] yep yep [21:02:06] there are a few over warninngs here and there [21:02:17] it is a bit of a pity that Gerrit failed there, cause we end up with broken code [21:02:55] but it does not seem to be the end of the world since I guess that branches is not going to be deployed in this state anytime soon [21:04:34] ejegg: I am heading to bed. Good luck on the upgrades! [21:05:55] thanks! [21:21:44] Hey all - anybody on releng have a minute for the best way to deal with a git reset --hard issue on prod? [21:24:24] sbassett: if there are local commits, those should either be okay to push off (checkout, not reset), or might be a local outdated state that are actually already submitted in which case a pull --rebase should address it [21:24:36] but one can fetch remote first to see which issue it is [21:24:55] or ask the person who left it that way (committer) to resolve it :) [21:25:12] So the issue is that a git reset --hard HEAD^ was accidentally run in /srv/mediawiki-staging/php-1.37.0-wmf.11. Do we just want to git fetch to restore that last commit? [21:25:57] I'm not sure we can, given submodules and sec patches [21:26:06] reflog should have the last known pointer though [21:26:16] Ok [21:26:24] then you can checkout that ref [21:26:41] and then resolve any dirty file/submodule changes after that one at a time [21:27:34] 10Release-Engineering-Team (Logspam), 10Platform Engineering, 10User-Ladsgroup, 10Wikimedia-production-error: PHP Notice: Undefined index: frameCount - https://phabricator.wikimedia.org/T285431 (10Ladsgroup) Okay, I found the problem. Not sure how to fix though. What you said was a great help actually but... [21:29:55] Krinkle: this is what I got from `git reflog` https://www.irccloud.com/pastebin/UaoIT5A9/ [21:30:44] and I was thinking about doing `git reset HEAD@{1}` [21:31:24] 10Release-Engineering-Team (Logspam), 10Platform Engineering, 10User-Ladsgroup, 10Wikimedia-production-error: PHP Notice: Undefined index: frameCount - https://phabricator.wikimedia.org/T285431 (10Ladsgroup) Okay, I have an idea. Let me try. [21:34:05] I think it was just the existing security patch (SECURITY: Prevent blocked users from purging pages) that likely got dropped. [21:34:27] So we can reapply that and I think we should be good, since it should already have been scapped out previously. [21:35:54] maryum: that works yeah, or use the sha1 hash in front of it (a bit less dynamic / racey) [21:36:07] I think we're ok now, but Maryum and I will leave a note at https://phabricator.wikimedia.org/T276237 for releng et al about this. [21:36:34] then you get the full state back, which is differnet from cherry-picking one commit, without worrying about anything else that may or may not have been there [22:12:20] 10Release-Engineering-Team (Logspam), 10Platform Engineering, 10Patch-For-Review, 10User-Ladsgroup, 10Wikimedia-production-error: PHP Notice: Undefined index: frameCount - https://phabricator.wikimedia.org/T285431 (10dduvall) I neglected to include this in the report, but a similar notice was occurring f... [22:15:56] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.31.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279725 (10DannyS712) [22:16:12] 10MediaWiki-Releasing, 10Security: Tracking bug for MediaWiki 1.31.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279726 (10DannyS712) [22:16:33] 10MediaWiki-Releasing, 10Security: Write and send pre-release announcements for MediaWiki 1.31.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279727 (10DannyS712) [22:16:45] 10MediaWiki-Releasing, 10Security: Write and send release announcements for MediaWiki 1.31.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279728 (10DannyS712) [22:16:56] 10MediaWiki-Releasing, 10Security: Update onwiki release notes for 1.31.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279729 (10DannyS712) [22:17:09] 10MediaWiki-Releasing, 10MW-1.37-notes (1.37.0-wmf.12; 2021-06-28), 10Security: Update HISTORY in master after 1.31.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279730 (10DannyS712) [22:17:50] 10MediaWiki-Releasing, 10Security: Tag 1.31.15/1.35.3/1.36.1 - https://phabricator.wikimedia.org/T279731 (10DannyS712) [22:18:04] 10MediaWiki-Releasing, 10Security: Obtain CVEs for 1.31.15/1.35.3/1.36.1 security releases - https://phabricator.wikimedia.org/T279732 (10DannyS712)