[00:49:47] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s: Add helm rollback functionality to scap - https://phabricator.wikimedia.org/T371069 (10RLazarus) 03NEW [00:51:39] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s: Add helm rollback functionality to scap - https://phabricator.wikimedia.org/T371069#10016831 (10RLazarus) [04:54:40] (03update) 10kharlan: Enable temp accounts by default [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/627 [07:34:26] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Increase disk size for GitLab test instance - https://phabricator.wikimedia.org/T369837#10017074 (10Jelto) [07:34:27] 10GitLab, 06collaboration-services: gitlab/devtools: send logs to a new disk - https://phabricator.wikimedia.org/T371066#10017075 (10Jelto) [07:43:14] 10Gerrit: Unchanged file with unresolved comment doesn't show up in file list - https://phabricator.wikimedia.org/T362269#10017102 (10ihurbain) Possibly related - I was complaining yesterday about Gerrit not showing me unresolved comments on a previous version of a file. So I tested a couple of things on /test/g... [07:56:38] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s: Add helm rollback functionality to scap - https://phabricator.wikimedia.org/T371069#10017126 (10akosiaris) [07:57:00] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s: Add helm rollback functionality to scap - https://phabricator.wikimedia.org/T371069#10017127 (10akosiaris) Good point, do we have some more information why the automatic rollback didn't happen/failed? [08:58:31] 10Release-Engineering-Team (Radar), 06collaboration-services, 06SRE, 06Traffic, 13Patch-For-Review: implement anti-abuse features for GitLab (Move GitLab behind the CDN) - https://phabricator.wikimedia.org/T366882#10017292 (10Jelto) [09:07:41] 10Release-Engineering-Team (Radar), 06collaboration-services, 06SRE, 06Traffic, 13Patch-For-Review: implement anti-abuse features for GitLab (Move GitLab behind the CDN) - https://phabricator.wikimedia.org/T366882#10017310 (10Jelto) >>! In T366882#10012178, @Dzahn wrote: > gitlab1003 and gitlab1004 are u... [10:14:26] GitLab needs a short maintenance at 11:00 UTC (in 45 minutes) [10:26:19] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T366961#10017524 (10Umherirrender) [10:30:59] 10Scap, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Evaluate the performance improvements brought in by prefetching MW images on WikiKube hosts - https://phabricator.wikimedia.org/T366778#10017533 (10akosiaris) No significant changes in the last month per {F56700235}, proceeding with the drop to 66% [10:37:21] 10Phabricator: Display info for new technical contributors in task comment field for new accounts - https://phabricator.wikimedia.org/T369982#10017552 (10Aklapper) 05Open→03Stalled p:05Triage→03Low [10:38:24] 10Phabricator: Explore displaying additional info in comment field for open Outreach related tasks - https://phabricator.wikimedia.org/T367500#10017555 (10Aklapper) I would not want to introduce more tags, the existing ones are already inconsistently used... [10:43:05] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 06Data-Platform-SRE, 10Wikidata, and 3 others: move commons-query.wikimedia.org and query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10017564 (10WMDE-leszek) Thanks @Jelto for setting up the Gitlab repository. I'... [11:06:04] GitLab upgrade done [11:57:19] !log temporarily starting deployment-deploy03 again so I can copy /root/secrets.txt (beta cluster logstash access) out of it [11:57:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:59:06] !log copied /root/secrets.txt from deployment-deploy03 to deployment-deploy04 [11:59:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:59:31] !log stopped deployment-deploy03 again [11:59:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:00:08] (also updated https://wikitech.wikimedia.org/wiki/Logstash#Beta_Cluster_Logstash) [12:02:53] 10Gerrit, 10VPS-project-Codesearch: Codesearch shows .md files as rendered - https://phabricator.wikimedia.org/T371092 (10Tgr) 03NEW [12:15:29] 10Gerrit, 10VPS-project-Codesearch: Codesearch shows .md files as rendered - https://phabricator.wikimedia.org/T371092#10017712 (10Tgr) Upstream issue is https://issues.gerritcodereview.com/issues/40014899. [13:02:26] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 06Data-Platform-SRE, 10Wikidata, and 3 others: move commons-query.wikimedia.org and query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10017771 (10Jelto) >>! In T350793#10017564, @WMDE-leszek wrote: > Thanks @Jelto... [13:20:18] (03PS1) 10Hashar: Remove .puppet-lint.rc [integration/config] - 10https://gerrit.wikimedia.org/r/1057208 (https://phabricator.wikimedia.org/T209361) [13:24:02] (03CR) 10Jforrester: [C:03+2] Remove .puppet-lint.rc [integration/config] - 10https://gerrit.wikimedia.org/r/1057208 (https://phabricator.wikimedia.org/T209361) (owner: 10Hashar) [13:25:11] (03Merged) 10jenkins-bot: Remove .puppet-lint.rc [integration/config] - 10https://gerrit.wikimedia.org/r/1057208 (https://phabricator.wikimedia.org/T209361) (owner: 10Hashar) [13:29:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar), 06collaboration-services: Decommission integration.mediawiki.org - https://phabricator.wikimedia.org/T361250#10017803 (10Dzahn) [13:29:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar), 06collaboration-services: Decommission integration.mediawiki.org - https://phabricator.wikimedia.org/T361250#10017804 (10Dzahn) 05In progress→03Resolved [13:50:06] (03merge) 10matmarex: Enable temp accounts by default [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/627 (owner: 10kharlan) [15:25:39] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s: Add helm rollback functionality to scap - https://phabricator.wikimedia.org/T371069#10018321 (10RLazarus) I haven't dug for logs, but workers were saturated and latency was up: https://grafana.wikimedia.org/goto/h67ix_uSg I'm assuming we were just failing hea... [15:57:43] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10MW-on-K8s: Add helm rollback functionality to scap - https://phabricator.wikimedia.org/T371069#10018541 (10thcipriani) [16:29:24] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T366960#10018727 (10dduvall) 05Open→03Resolved [17:27:24] (03open) 10taavi: runner: Handle unknown stylelint rules [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/57 [17:28:06] (03update) 10taavi: runner: Handle unknown stylelint rules [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/57 [17:30:15] (03merge) 10taavi: runner: Handle unknown stylelint rules [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/57 [17:33:11] (03open) 10brennen: Update README to reflect current state [repos/phabricator/tools] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/tools/-/merge_requests/5 (https://phabricator.wikimedia.org/T355574) [17:33:17] (03merge) 10brennen: Update README to reflect current state [repos/phabricator/tools] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/tools/-/merge_requests/5 (https://phabricator.wikimedia.org/T355574) [17:34:27] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 13Patch-For-Review, 07Python3-Porting: Many (all?) of the phabricator/tools scripts are in Python 2 - https://phabricator.wikimedia.org/T355574#10018963 (10brennen) 05Open→03Resolved a:03brennen [18:04:03] 10Phabricator: Do not autolink shard names (e.g. s1) as Phabricator Spaces - https://phabricator.wikimedia.org/T371131 (10Aklapper) 03NEW [18:08:20] 10Phabricator: Do not autolink shard names (e.g. s1) as Phabricator Spaces - https://phabricator.wikimedia.org/T371131#10019040 (10Dzahn) +1 Oh, yea I have often noticed this in the past. Thanks for creating this. [18:25:55] 10Phabricator: Do not autolink shard names (e.g. s1) as Phabricator Spaces - https://phabricator.wikimedia.org/T371131#10019092 (10Aklapper) I only stumbled upon this code today :-/ [18:57:42] 10Phabricator, 06Release-Engineering-Team: Do not autolink shard names (e.g. s1) as Phabricator Spaces - https://phabricator.wikimedia.org/T371131#10019203 (10brennen) [19:11:28] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10Wikimedia-Apache-configuration, and 3 others: Apache 2.4.61 throws a 403 Forbidden for links containing %3F - https://phabricator.wikimedia.org/T370110#10019291 (10brennen) Doesn't look like a lot else in codesearch... [21:42:09] 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team (Radar), 07Upstream: Investigate whether issues, operations, wikis, etc. can be disabled globally on GitLab - https://phabricator.wikimedia.org/T264231#10019675 (10thcipriani) 05Stalled→03Resolved a:03brennen Now that we have https:/... [21:44:53] 10Continuous-Integration-Config, 06MediaWiki-Platform-Team, 10MediaWiki-Vendor, 10Quibble, 07Developer Productivity: mediawiki/core and mediawiki/vendor both skip composer.lock checks - https://phabricator.wikimedia.org/T370380#10019681 (10Krinkle) p:05Triage→03Medium a:03Krinkle [22:14:15] 10Continuous-Integration-Config, 06MediaWiki-Platform-Team, 10MediaWiki-Vendor, 10Quibble, and 2 others: mediawiki/core and mediawiki/vendor both skip composer.lock checks - https://phabricator.wikimedia.org/T370380#10019779 (10Krinkle) >>! In T370380#10019764, @gerritbot wrote: > Change #1057272 had a rel... [22:19:22] hashar: I wonder if we can trigger both by ENV instead of skipping the check in 2 diferent ways in two different places [23:36:56] (03open) 10catrope: Chart: Run npm build [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/628