[00:21:37] <DannyS712>	 ^ was me fixing "1.32.15" to "1.31.15" for the various release tasks
[00:27:38] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap, 10User-Ladsgroup: image sizes not displayed on beta - https://phabricator.wikimedia.org/T285125 (10dpifke) I tried downgrading scap in beta to match the version in prod (by running `sudo apt-get install scap=3.17.1-1` on `deployment-deploy0...
[00:38:13] <wikibugs>	 10Release-Engineering-Team (Logspam), 10Platform Engineering, 10MW-1.37-notes (1.37.0-wmf.11; 2021-06-21), 10Patch-For-Review, and 2 others: PHP Notice: Undefined index: frameCount - https://phabricator.wikimedia.org/T285431 (10Ladsgroup) I made the patch but it's too late for me to deploy, anyone feeling...
[01:11:16] <Krinkle>	 !log deployment-memc08 and -memc09: apt-get install memkeys (already installed on deployment-mediawiki11)
[01:11:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[02:13:11] <wikibugs>	 (03CR) 10Eileen: "I've just been hitting some CI issues & realised it's because this is still not merged. I'm just wondering where we got to on this." [integration/config] - 10https://gerrit.wikimedia.org/r/689947 (https://phabricator.wikimedia.org/T277500) (owner: 10Jforrester)
[05:20:12] <wikibugs>	 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T281152 (10DannyS712) >>! In T281152#7154103, @Krinkle wrote: > ##### Risky Patch!  🚂🔥 >  > * **Change**:  https://gerrit.wikimedia.org/r/c/mediawiki/core/+/673311/ (T2777...
[06:58:53] <wikibugs>	 10MediaWiki-Codesniffer, 10phan, 10MediaWiki-General, 10MediaWiki-extensions-General, and 2 others: $maintClass in maintenance class files should be set with ::class name resolution - https://phabricator.wikimedia.org/T279267 (10Sahilgrewalhere) a:05Sahilgrewalhere→03None
[08:12:57] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap, 10WMDE-TechWish-Sprint-2021-06-23: Localisation cache is broken in beta - https://phabricator.wikimedia.org/T285345 (10thiemowmde)
[08:42:15] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Lets do the magic and we can then +2 https://gerrit.wikimedia.org/r/c/wikimedia/portals/deploy/+/700835" [integration/config] - 10https://gerrit.wikimedia.org/r/700911 (https://phabricator.wikimedia.org/T285239) (owner: 10Hashar)
[08:44:16] <wikibugs>	 (03Merged) 10jenkins-bot: Switch portals/deploy to NodeJS 12 [integration/config] - 10https://gerrit.wikimedia.org/r/700911 (https://phabricator.wikimedia.org/T285239) (owner: 10Hashar)
[08:48:08] <kostajh>	 hashar: guten morgen! do you mind deploying https://gerrit.wikimedia.org/r/c/integration/config/+/701050, if it looks correct, that is?
[08:50:24] <hashar>	 kostajh: sure ;)
[08:50:55] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] zuul: Add Apache/Selenium job to experimental group for GrowthExperiments and VE [integration/config] - 10https://gerrit.wikimedia.org/r/701050 (https://phabricator.wikimedia.org/T276428) (owner: 10Kosta Harlan)
[08:52:07] <wikibugs>	 (03Merged) 10jenkins-bot: zuul: Add Apache/Selenium job to experimental group for GrowthExperiments and VE [integration/config] - 10https://gerrit.wikimedia.org/r/701050 (https://phabricator.wikimedia.org/T276428) (owner: 10Kosta Harlan)
[08:52:46] <hashar>	 kostajh: deployed!
[08:52:51] <kostajh>	 hashar: thank you!
[09:04:08] <wikibugs>	 10Release-Engineering-Team (Logspam), 10Platform Engineering, 10MW-1.37-notes (1.37.0-wmf.12; 2021-06-28), 10Patch-For-Review, and 2 others: PHP Notice: Undefined index: frameCount - https://phabricator.wikimedia.org/T285431 (10Ladsgroup) This is handled on dev side, we can roll the train forward IMO.
[09:06:24] <wikibugs>	 (03PS1) 10Hashar: Inject LOG_DIR for node12-browser jobs [integration/config] - 10https://gerrit.wikimedia.org/r/701344
[09:07:21] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Inject LOG_DIR for node12-browser jobs [integration/config] - 10https://gerrit.wikimedia.org/r/701344 (owner: 10Hashar)
[09:09:04] <wikibugs>	 (03Merged) 10jenkins-bot: Inject LOG_DIR for node12-browser jobs [integration/config] - 10https://gerrit.wikimedia.org/r/701344 (owner: 10Hashar)
[09:13:40] <kostajh>	 hashar: yay, got VE working in a CI build
[09:21:24] <wikibugs>	 (03PS1) 10Kosta Harlan: Add parsoid dependency to VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/701346 (https://phabricator.wikimedia.org/T227352)
[09:22:27] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add parsoid dependency to VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/701346 (https://phabricator.wikimedia.org/T227352) (owner: 10Kosta Harlan)
[09:33:56] <wikibugs>	 (03PS2) 10Kosta Harlan: Add parsoid dependency to VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/701346 (https://phabricator.wikimedia.org/T227352)
[10:13:27] <kostajh>	 Daimona: I saw some strange errors for AbuseFilter selenium tests in an experimental build that uses Apache, was wondering if you might have seen it before and know what's going on
[10:14:05] <Daimona>	 Hmmm do you have any link? I am only aware of a single, intermittent error which has no obvious cause
[10:14:07] <kostajh>	 https://integration.wikimedia.org/ci/job/wmf-quibble-apache-selenium-php72-docker/1809/console has the errors, and in particular for AbuseFilter it's "MWException: HTMLForm: no submit callback provided. Use setSubmitCallback() to set one"
[10:14:55] <Daimona>	 Ouch, no, this one is new to me. Checking now.
[10:16:19] <kostajh>	 It seems unrelated to the Apache change
[10:19:19] <Daimona>	 It's weird though, submitting the form works for me locally and in prod
[10:22:12] <kostajh>	 yeah, same here
[10:23:31] <Daimona>	 OK I was able to reproduce
[10:24:02] <Daimona>	 The form action is set to the URL of Special:AbuseFilter/new, which locally resolves to //localhost/pedia/index.php/Special:AbuseFilter/new
[10:24:33] <Daimona>	 But I can see from the build artifacts that it's trying 127.0.0.1:9413/index.php?title=Special:AbuseFilter/new
[10:25:07] <Daimona>	 That is, it's using the "title=..." variant. If I change the form action to that variant locally it fails
[10:26:49] <Daimona>	 So the immediate cause is likely to be some config difference related to URIs. OTOH, it's not clear why the "title" variant wants triggers the HTMLForm callback and the other one doesn't
[10:29:34] <kostajh>	 oh. weird
[10:33:17] <Daimona>	 Now looking... The form has a hidden "title" input whose value is "Special:AbuseFilter/import". That's passed to the server and it makes SpecialPageFactory give us the wrong page
[10:35:08] <Daimona>	 So I think what happens is the hidden input takes precedence over the query string and it sends the user back to the import page, which is wrong.
[10:37:46] <Daimona>	 ^Confirming that. The problem is, that "title" input is set by HTMLForm for all POST forms and there doesn't seem a way to disable it. I'm not even sure what its purpose is. And as we're seeing here, it will conflict with the form action if full URLs are resolved with the "title" param. I think this is an HTMLForm bug?!
[10:41:05] <kostajh>	 seems so, yeah
[10:41:15] <Daimona>	 Maybe this fix should've been applied to the POST case, too? https://gerrit.wikimedia.org/r/c/mediawiki/core/+/3680
[10:41:59] <kostajh>	 not sure
[10:43:36] <Daimona>	 I'm not sure either, but digging through git blame, it doesn't seem unreasonable to check whether short links are enabled before setting a title parameter.
[10:44:10] <kostajh>	 I don't know why this is only getting triggered in the apache variant; the php built in server would also be using index.php?title
[10:44:15] <Daimona>	 (Also, that particular patch is not entirely related, it was just switching from one global to another. The real point is checking what kind of links are used by the wiki)
[10:45:58] <Daimona>	 It doesn't seem to be the case https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/100532/artifact/log/When-importing-a-filter-it-should-redirect-to-ViewEdit-after-submission-2021-06-24T04-05-48-516Z.mp4
[10:53:50] <Daimona>	 Created https://phabricator.wikimedia.org/T285464
[11:15:42] <icinga-wm>	 PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[11:16:11] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap, 10User-Ladsgroup: image sizes not displayed on beta - https://phabricator.wikimedia.org/T285125 (10Jpita) a similar issue is happening here  https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version version: 1.37.0-alpha (1f66af0) 09:37, 2...
[11:23:52] <wikibugs>	 (03PS1) 10Kormat: jjb: Support pcc hosts: comments [integration/config] - 10https://gerrit.wikimedia.org/r/701370
[11:27:11] <wikibugs>	 (03CR) 10Kormat: "An example testcase: https://phabricator.wikimedia.org/P16724" [integration/config] - 10https://gerrit.wikimedia.org/r/701370 (owner: 10Kormat)
[11:43:53] <wikibugs>	 (03CR) 10Jbond: "LGTM see comments inline" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/701370 (owner: 10Kormat)
[12:16:27] <icinga-wm>	 RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[12:29:30] <jgleeson>	 hey there releng folks. Would someone be able to +2 a patch to our CI config here https://gerrit.wikimedia.org/r/c/integration/config/+/689947 ?
[12:30:31] <wikibugs>	 (03PS2) 10Hashar: operations-puppet: Refresh to update gems [integration/config] - 10https://gerrit.wikimedia.org/r/701079 (owner: 10Jbond)
[12:31:18] <jgleeson>	 James_F: found a bug when testing the first patch that we've since fixed in https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/692693/ but we can't +2 that patch until the aforementioned patch is merged otherwise it'll break CI for our CRM build.
[12:34:15] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] operations-puppet: Refresh to update gems [integration/config] - 10https://gerrit.wikimedia.org/r/701079 (owner: 10Jbond)
[12:35:25] <wikibugs>	 (03Merged) 10jenkins-bot: operations-puppet: Refresh to update gems [integration/config] - 10https://gerrit.wikimedia.org/r/701079 (owner: 10Jbond)
[13:17:06] <jgleeson>	 hashar: would you be able to +2 a CI patch for us over in fundraising-tech? https://gerrit.wikimedia.org/r/c/integration/config/+/689947
[13:34:31] <wikibugs>	 (03PS2) 10Kormat: jjb: Support pcc hosts: comments [integration/config] - 10https://gerrit.wikimedia.org/r/701370
[13:40:44] <hashar>	 jgleeson: yeah for sure
[13:40:44] <wikibugs>	 (03CR) 10Jbond: [C: 03+1] "LGTM thanks <3" [integration/config] - 10https://gerrit.wikimedia.org/r/701370 (owner: 10Kormat)
[13:42:44] <hashar>	 jgleeson: hmm no we can't do that :D
[13:47:22] <hashar>	 and of course they don't publish it to packagist.org
[13:47:23] <wikibugs>	 (03CR) 10Kormat: jjb: Support pcc hosts: comments (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/701370 (owner: 10Kormat)
[13:50:06] <jgleeson|mobile>	 hashar how come?
[13:51:44] <hashar>	 jgleeson|mobile: a couple issues really ;D
[13:51:57] <hashar>	 that clones the repo on the Jenkins host as the jenins user
[13:52:08] <hashar>	 which would mismatch with the user used in the docker containers (nobody)
[13:52:30] <hashar>	 so that is not too hard to change the command to use a container to run git into
[13:52:58] <hashar>	 the other issue is that it just grab whatever is the latest commit of civicrm-buildkit which sounds like it is out of our control
[13:53:36] <hashar>	 so maybe we can keep our civicrm-buildkit mirror on Gerrit and update it from time to time?
[13:56:26] <wikibugs>	 (03CR) 10Jbond: [C: 03+1] jjb: Support pcc hosts: comments (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/701370 (owner: 10Kormat)
[13:57:30] <jgleeson|mobile>	 ah ok the reason we've updating the is to remove the dependency on maintaining our own fork as the upstream is updated frequently and it is a maintenance headache
[13:58:08] <jgleeson|mobile>	 we're updating that*
[14:02:50] <hashar>	 jgleeson|mobile: ok got it. But that means you get exposed to whatever change might happen in upstream repo
[14:02:55] <hashar>	 which might break stuff here and there
[14:59:29] <hashar>	 $(...).click is not a function
[14:59:34] <hashar>	 I am giving up :-(
[15:00:32] <wikibugs>	 (03PS1) 10Hashar: jjb: update operations-puppet image [integration/config] - 10https://gerrit.wikimedia.org/r/701395
[15:01:48] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] jjb: update operations-puppet image [integration/config] - 10https://gerrit.wikimedia.org/r/701395 (owner: 10Hashar)
[15:03:59] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: update operations-puppet image [integration/config] - 10https://gerrit.wikimedia.org/r/701395 (owner: 10Hashar)
[15:07:05] <jgleeson>	 ok hashar leave it with me and I'll discuss with it the rest of fr-tech to see how best to proceed. thanks for reviewing the patch! 
[15:07:11] <jgleeson>	 ejegg: ^^^
[15:07:41] <hashar>	 jgleeson: yaeh and I should comment on the change
[15:07:44] <ejegg>	 reading backscroll
[15:10:00] <ejegg>	 so buildkit (even the locally mirrored version) pulls down a lot of other tools to do its job, and we're already exposed to upstream changes in those tools
[15:10:41] <ejegg>	 would it somehow be better to just have a minimal local script that pulls down the latest buildkit each time?
[15:13:59] <ejegg>	 err, or have the dockerfile pull from the remote repo instead of our local?
[15:16:35] <wikibugs>	 (03CR) 10Hashar: jjb: [wikimedia-fundraising-civicrm-docker] Use upstream civicrm-buildkit (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/689947 (https://phabricator.wikimedia.org/T277500) (owner: 10Jforrester)
[15:16:51] <hashar>	 jgleeson: ejegg I wrote a summary on the change ^
[15:16:58] <hashar>	 and I gotta escape cause of kids dinner etc
[15:17:08] <hashar>	 but i will be back later this evening
[15:22:06] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #25906: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/25906/
[16:03:10] <jgleeson>	 thanks hashar !
[16:18:47] <wmf-insecte>	 Yippee, build fixed!
[16:18:47] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #25907: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/25907/
[18:37:40] <jgleeson>	 hello releng wizards. is there a template you folks used to set the timezone dashboard thingy up here https://timerelengteam.toolforge.org/
[18:38:06] <jgleeson>	 we have a mix of timezones over on fundraising-tech and something like that would be useful for us too
[18:39:10] <jgleeson>	 Looks like some petty straightforward JS is powering it. I guess we could borrow that
[19:03:20] <greg-g>	 jgleeson: yeah, feel free to copy/paste :) th.cipriani built it and is out this week
[19:04:16] <jgleeson>	 awesome, thanks much!
[19:04:25] <legoktm>	 the tool links to https://github.com/thcipriani/releng-junk as the source
[20:40:33] <wikibugs>	 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T281152 (10dduvall)
[20:40:48] <wikibugs>	 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T281152 (10dduvall) 05Open→03Resolved
[20:41:15] <wikibugs>	 10Release-Engineering-Team (Logspam), 10Platform Engineering, 10MW-1.37-notes (1.37.0-wmf.11; 2021-06-21), 10Patch-For-Review, and 2 others: PHP Notice: Undefined index: frameCount - https://phabricator.wikimedia.org/T285431 (10dduvall) 05Open→03Resolved Re-rolled to group1 at 1700 and I've seen no not...