[06:23:33] !log devtools: set `profile::phabricator::main::dumps_rsync_clients: []` project wide to fix up Puppet. Settings got moved to a `role` ( https://gerrit.wikimedia.org/r/c/operations/puppet/+/842875 | T313360 ) [06:23:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:23:35] T313360: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 [06:24:40] !log devtools: `rmdir /etc/envoy/clusters.d /etc/envoy/listeners.d` [06:24:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:24:58] !log devtools: phabricator-prod-1001: `rmdir /etc/envoy/clusters.d /etc/envoy/listeners.d` [06:24:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:36:50] 10Phabricator, 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ): Puppet failure on phabricator-stage-1001.devtools - https://phabricator.wikimedia.org/T299997 (10hashar) The instance was created in January 2020 by @Dzahn. It has been failing Puppet for a year now and apparently nobody has a use case for it. I... [08:05:16] hashar: more specifically, I moved shared setup code for api-testing and Selenium to https://github.com/wikimedia/mediawiki-extensions-GrowthExperiments/blob/master/tests/LocalSettingsSetup.cjs, and make use of that file in api-tesing and Selenium setup functions [09:32:33] 10Phabricator (Search): Cannot search by Task title in some cases - https://phabricator.wikimedia.org/T305065 (10valerio.bozzolan) [09:37:49] 10Phabricator (Search): Cannot find some Tasks using simple search - https://phabricator.wikimedia.org/T305065 (10valerio.bozzolan) [13:02:38] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-api-int helmfile deployment - https://phabricator.wikimedia.org/T321895 (10Clement_Goubert) [13:03:12] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-api-ext helmfile deployment - https://phabricator.wikimedia.org/T321896 (10Clement_Goubert) [13:03:39] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-jobrunner helmfile deployment - https://phabricator.wikimedia.org/T321897 (10Clement_Goubert) [13:04:15] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-videoscaler helmfile deployment - https://phabricator.wikimedia.org/T321899 (10Clement_Goubert) [13:04:38] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Create mw-web helmfile deployment - https://phabricator.wikimedia.org/T321900 (10Clement_Goubert) [13:06:24] 10Phabricator (Search), 10Discovery-Search, 10Elasticsearch: Cannot find some Tasks using simple search - https://phabricator.wikimedia.org/T305065 (10hashar) #elasticsearch since Phabricator uses that as backend: ` lang=yaml,name=hieradata/role/eqiad/phabricator.yaml phabricator_cluster_search: - type: 'e... [13:15:23] 10Phabricator (Search): Cannot find some Tasks using simple search - https://phabricator.wikimedia.org/T305065 (10hashar) We had a task to verify compatibility with ElasticSearch 7.10 then T303445#7807237 states Phabricator uses MySQL. From https://phabricator.wikimedia.org/config/cluster/search/ : {F35649258 s... [13:26:39] 10Phabricator (Search): No search results for recently created task - https://phabricator.wikimedia.org/T263836 (10hashar) 05Open→03Resolved a:03hashar `title:growth title:hebrew` (without `AND`) gives me two tasks: * `T263794` Enable **Growth** features by default in the **Hebrew** Wikipedia * `T255020`... [13:32:02] 10Phabricator (Search): Substring in title not found in Phabricator Global Search - https://phabricator.wikimedia.org/T252149 (10hashar) 05Open→03Resolved a:03hashar Search for `EchoPerUserBlacklist` From https://secure.phabricator.com/book/phabricator/article/search/ > Substrings: Normally, query terms a... [13:33:50] 10Beta-Cluster-Infrastructure, 10Discovery-Search: Translate ES ansible playbook to cumin cookbook - https://phabricator.wikimedia.org/T302736 (10hashar) I have removed #phab-search . According to closed {T301954} the #beta-cluster-infrastructure ElasticSearch has been upgraded so I guess this task is no more... [13:38:21] 10Phabricator (Search): "Tags" field in Maniphest Advanced Search does not offer MediaWiki-API in proposals when using "any" prefix - https://phabricator.wikimedia.org/T218103 (10hashar) `any(mediawiki-action-api)` works and is the canonical name of the project. `#mediawiki-api` is an additional hashtag. If I l... [14:14:40] !log delete deployment-db07 and deployment-db08 [14:14:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:51:34] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Scap, 10Patch-For-Review: Strictly lock scap requirements dependencies - https://phabricator.wikimedia.org/T316755 (10thcipriani) p:05Triage→03Medium [14:56:37] 10Continuous-Integration-Config, 10Release-Engineering-Team (Blocking 🧱), 10SRE, 10serviceops, 10Test-Coverage: Add pcov PHP extension to wikimedia apt so it can be used in Wikimedia CI - https://phabricator.wikimedia.org/T243847 (10thcipriani) [15:00:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10SRE, 10Patch-For-Review: The python-build images regenerate wheels even when matching ones are already available - https://phabricator.wikimedia.org/T259611 (10thcipriani) p:05Medium→03Low a:03hashar [15:01:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10SRE, 10Patch-For-Review: The python-build images regenerate wheels even when matching ones are already available - https://phabricator.wikimedia.org/T259611 (10thcipriani) @hashar is this waiting on review, or are y... [15:05:15] <_joe_> thcipriani: I noticed you use the brick emoji in your boards; it's the symbol italian alt-right uses to identify themselves on twitter :P [15:05:35] haha oh man [15:05:39] no emoji is safe [15:05:48] <_joe_> yeah I mean there's no chance you can be confused [15:07:58] Why a brick? [15:08:20] It's something idiotic isn't it? [15:10:24] <_joe_> claime: tbh I read the explanation once but I forgot [15:10:41] <_joe_> I have no cache space for their bs [15:10:43] "Knowledge is power, but this is so dumb I'll forget it" [15:10:45] :p [15:10:50] <_joe_> exactly [15:11:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Blocking 🧱), 10Quibble, 10Wikidata, 10Patch-For-Review: Terminating MySQL takes several minutes in (Wikibase?) CI jobs - https://phabricator.wikimedia.org/T265615 (10thcipriani) 05Open→03Resolved Calling this resolved. I took a look... [15:19:20] 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾), 10Scap, 10Patch-For-Review: Update scap documentation and other references for new GitLab location - https://phabricator.wikimedia.org/T321847 (10dancy) https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/10 was merged [15:39:42] (03PS1) 10David Caro: tox-poetry-buster: upgrade poetry install script [integration/config] - 10https://gerrit.wikimedia.org/r/850515 [15:41:06] (03CR) 10David Caro: "Tested locally running the image:" [integration/config] - 10https://gerrit.wikimedia.org/r/850515 (owner: 10David Caro) [15:41:30] (03CR) 10CI reject: [V: 04-1] tox-poetry-buster: upgrade poetry install script [integration/config] - 10https://gerrit.wikimedia.org/r/850515 (owner: 10David Caro) [15:42:54] (03CR) 10David Caro: "Dummy flake8 xd" [integration/config] - 10https://gerrit.wikimedia.org/r/850515 (owner: 10David Caro) [15:44:23] (03PS2) 10David Caro: tox-poetry-buster: upgrade poetry install script [integration/config] - 10https://gerrit.wikimedia.org/r/850515 [15:45:51] (03PS3) 10David Caro: tox-poetry-buster: upgrade poetry install script and poetry to 1.2.2 [integration/config] - 10https://gerrit.wikimedia.org/r/850515 [15:47:22] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T314191 (10thcipriani) 05Open→03Resolved [15:47:35] (03CR) 10CI reject: [V: 04-1] tox-poetry-buster: upgrade poetry install script and poetry to 1.2.2 [integration/config] - 10https://gerrit.wikimedia.org/r/850515 (owner: 10David Caro) [15:48:10] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T320511 (10thcipriani) 05Open→03Resolved [15:49:46] (03PS4) 10David Caro: tox-poetry-buster: upgrade poetry install script [integration/config] - 10https://gerrit.wikimedia.org/r/850515 [15:55:33] (03PS5) 10David Caro: tox-poetry-buster: upgrade poetry install script and upgrade to 1.2.2 [integration/config] - 10https://gerrit.wikimedia.org/r/850515 [15:58:33] 10Continuous-Integration-Infrastructure: poetry is not working in tox-poetry-buster:0.0.2 - https://phabricator.wikimedia.org/T321915 (10fnegri) [16:07:48] 10Continuous-Integration-Infrastructure: poetry is not working in tox-poetry-buster:0.0.2 - https://phabricator.wikimedia.org/T321915 (10fnegri) [16:46:01] 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾), 10Scap, 10Patch-For-Review: Update scap documentation and other references for new GitLab location - https://phabricator.wikimedia.org/T321847 (10dancy) https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/12 [17:10:50] (03PS1) 10Subramanya Sastry: Tweak CSS styles based on script output + previous patches [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/850526 [19:26:15] 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾), 10Scap, 10Patch-For-Review: Update scap documentation and other references for new GitLab location - https://phabricator.wikimedia.org/T321847 (10dancy) I ran this on deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud in /srv/deployment/scap: `... [20:01:26] the joyce of late evening hacking: `TypeError: str expected, not str` [20:03:08] (03PS1) 10Umherirrender: Zuul: [mediawiki/extensions/NaylorAMS] Add [integration/config] - 10https://gerrit.wikimedia.org/r/850538 [20:05:20] (03CR) 10Hashar: Zuul: [mediawiki/extensions/NaylorAMS] Add (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/850538 (owner: 10Umherirrender) [20:06:10] (03PS2) 10Umherirrender: Zuul: [mediawiki/extensions/NaylorAMS] Add [integration/config] - 10https://gerrit.wikimedia.org/r/850538 [20:06:32] (03PS3) 10Hashar: Zuul: [mediawiki/extensions/NaylorAMS] Add [integration/config] - 10https://gerrit.wikimedia.org/r/850538 (owner: 10Umherirrender) [20:06:47] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/NaylorAMS] Add [integration/config] - 10https://gerrit.wikimedia.org/r/850538 (owner: 10Umherirrender) [20:06:50] (03CR) 10Umherirrender: [C: 03+1] Zuul: [mediawiki/extensions/NaylorAMS] Add (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/850538 (owner: 10Umherirrender) [20:09:49] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/NaylorAMS] Add [integration/config] - 10https://gerrit.wikimedia.org/r/850538 (owner: 10Umherirrender) [20:30:22] How exactly does the beta cluster update itself and is there a way to manually trigger a deploy or debug when a patch might get deployed? [20:30:25] Right now I'm looking to test a patch but it hasn't showed up there and it's been over an hour. [20:31:10] (03CR) 10Hashar: [C: 03+2] "Deployed, sorry for the delay" [integration/config] - 10https://gerrit.wikimedia.org/r/850538 (owner: 10Umherirrender) [20:31:24] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10serviceops, 10serviceops-collab, 10Patch-For-Review: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) >>! In T313360#8351748, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-releng),... [20:32:31] (03PS1) 10Umherirrender: Zuul: [mediawiki/extensions/HTMLPurifier] Add [integration/config] - 10https://gerrit.wikimedia.org/r/850544 [20:33:38] Jdlrobson, there is a jenkins job triggered every ten minutes, see https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated [20:33:52] you can manually trigger it at https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/, but it seems to be running normaly [20:37:01] zabe: :-] [20:37:09] thanks for helping! [20:37:56] and the documentation is at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated (poke Jdlrobson [20:38:42] and https://integration.wikimedia.org/ci/view/Beta/ for an up-to-date status [20:42:03] (03CR) 10Arlolra: [C: 03+2] Use sup.reference instead of sup.mw-ref (courtesy T265930) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/848554 (owner: 10Subramanya Sastry) [20:42:27] (03CR) 10Arlolra: [C: 03+2] Tweak es/fr/hu CSS based on site messages / Common.css [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/849662 (owner: 10Subramanya Sastry) [20:42:35] (03CR) 10Arlolra: [C: 03+2] Remove default styles that are now shipped by Cite [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/850284 (owner: 10Subramanya Sastry) [20:42:52] (03Merged) 10jenkins-bot: Use sup.reference instead of sup.mw-ref (courtesy T265930) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/848554 (owner: 10Subramanya Sastry) [20:42:58] (03CR) 10Arlolra: [C: 03+2] Tweak CSS styles based on script output + previous patches [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/850526 (owner: 10Subramanya Sastry) [20:43:10] (03Merged) 10jenkins-bot: Tweak es/fr/hu CSS based on site messages / Common.css [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/849662 (owner: 10Subramanya Sastry) [20:43:18] (03Merged) 10jenkins-bot: Remove default styles that are now shipped by Cite [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/850284 (owner: 10Subramanya Sastry) [20:44:00] (03Merged) 10jenkins-bot: Tweak CSS styles based on script output + previous patches [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/850526 (owner: 10Subramanya Sastry) [20:52:37] hashar: zabe Thanks for the docs. It seems like ReadingList is not running the latest code. I don't see https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ReadingLists/+/850525 there [20:52:44] and Special:Version suggests it deployed the patch before [20:58:02] the new code does exist on the appservers [20:58:03] https://phabricator.wikimedia.org/P37041 [21:02:12] also based on time the change got merged and then crawling builds of the Jenkins job which updates beta [21:02:12] https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/415322/console [21:02:20] it lists the commit :) [21:02:56] I am off for the week-end [21:06:03] Farewell (again) [21:10:14] 🎃 [21:54:10] 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾), 10Scap, 10serviceops-collab: Set up equivalent of the scap-tox-publish job in scap's .gitlab-ci.yml - https://phabricator.wikimedia.org/T321629 (10Dzahn) Changes have been deployed as above and discussion on Gerrit. Confirmed that both firewall rules... [21:59:39] 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾), 10Scap, 10serviceops-collab: Set up equivalent of the scap-tox-publish job in scap's .gitlab-ci.yml - https://phabricator.wikimedia.org/T321629 (10Dzahn) @dancy @Jelto I went to gitlab-runner1002.eqiad.wmnet (what the first IP in the list resolved to)... [22:15:28] seems like I'm just having a bad day :) Thanks for the help! [22:16:39] 10Phabricator, 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ): Puppet failure on phabricator-stage-1001.devtools - https://phabricator.wikimedia.org/T299997 (10Dzahn) @hashar The idea was once that it's 2 different use cases. One should be always "like production" (to be able to test changes before merging in...