[08:08:32] 10Deployments, 06Release-Engineering-Team, 06SRE: Reduce automatic messages on #wikimedia-operations - https://phabricator.wikimedia.org/T384804 (10hashar) 03NEW [10:09:27] 10Deployments, 06Release-Engineering-Team, 06SRE, 13Patch-For-Review: Reduce automatic messages on #wikimedia-operations - https://phabricator.wikimedia.org/T384804#10496117 (10Volans) > I believe SRE are instead using their own private channel. It's `#wikimedia-sre` and it's a public channel (as mention... [10:18:59] 10Deployments, 06Release-Engineering-Team, 06SRE, 13Patch-For-Review: Reduce automatic messages on #wikimedia-operations - https://phabricator.wikimedia.org/T384804#10496179 (10hashar) [14:26:59] 10GitLab (Infrastructure), 06collaboration-services: GitLab OpenSSL 3 upgrade in 17.7 - https://phabricator.wikimedia.org/T379598#10497101 (10Jelto) Thanks for updating the `gitlab-ce` package @Dzahn. The package was installed automatically on the test instance by unattended upgrades: ` 2025-01-25 06:00:28,22... [14:27:29] 10GitLab (Infrastructure), 06collaboration-services: GitLab OpenSSL 3 upgrade in 17.7 - https://phabricator.wikimedia.org/T379598#10497102 (10Jelto) [14:34:06] 10WikimediaDebug, 07Wikimedia-production-error: On-demand Excimer profiling does not work (Jan 27th, 2025) - https://phabricator.wikimedia.org/T384836 (10mszabo) 03NEW [14:35:10] 10WikimediaDebug, 07Wikimedia-production-error: On-demand Excimer profiling does not work (Jan 27th, 2025) - https://phabricator.wikimedia.org/T384836#10497147 (10mszabo) The service seems to be unreachable from deploy nodes as well both in eqiad and codfw: ` $ curl --connect-timeout 10 -sv https://performanc... [14:53:28] 10WikimediaDebug, 07Wikimedia-production-error: On-demand Excimer profiling does not work (Jan 27th, 2025) - https://phabricator.wikimedia.org/T384836#10497244 (10mszabo) Looks to have been caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1103318 which now blocks all non-CDN traffic to these hosts. [14:59:36] 10WikimediaDebug, 13Patch-For-Review, 07Wikimedia-production-error: On-demand Excimer profiling does not work (Jan 27th, 2025) - https://phabricator.wikimedia.org/T384836#10497278 (10Joe) a:03Joe [15:06:13] 10WikimediaDebug, 13Patch-For-Review, 07Wikimedia-production-error: On-demand Excimer profiling does not work (Jan 27th, 2025) - https://phabricator.wikimedia.org/T384836#10497313 (10Joe) 05Open→03Resolved [15:48:31] 10Deployments, 06Release-Engineering-Team, 06SRE, 13Patch-For-Review: Reduce automatic messages on #wikimedia-operations - https://phabricator.wikimedia.org/T384804#10497480 (10Dzahn) Whatever we end up doing, let's resist the temptation to create yet another "-feed" channel (that few look at) because that... [16:54:05] is there a phabricator task for tracking when builds just randomly hang for no apparent reason? [16:54:13] just happened at https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-selenium/67100/console and I think there was another one a week or two ago [16:54:38] so far it seems rare enough that I’m okay with leaving it alone, but if it’s already tracked I thought I might as well add this one to the list somewhere :) [17:07:11] Lucas_WMDE: I noticed that during the week of Jan 13-17 some people force killed Selenium builds [17:07:39] #ci-test-error is the catch up tag for those failures https://phabricator.wikimedia.org/tag/ci-test-error/ [17:08:14] as to what is blocked or waiting, I have no idea [17:08:43] https://phabricator.wikimedia.org/T320283 looks like the closest task I could find on that board [17:08:44] 00:11:39.062 INFO:quibble.commands:Running webdriver test in /workspace/src/extensions/IPInfo [17:08:44] 00:42:39.539 Build was aborted [17:09:11] and presumably it does not manage to finish the `npm ci` installation [17:11:48] I guess when Qiubble receives a SIGTERM we could dump the list of processes [17:11:54] in the same process group [17:12:08] or well maybe Jenkins should do that for us :) [18:10:03] maintenance-disconnect-full-disks build 670818 integration-agent-docker-1051 (/: 27%, /srv: 100%, /var/lib/docker: 30%): OFFLINE due to disk space [18:14:27] :( [18:15:03] maintenance-disconnect-full-disks build 670819 integration-agent-docker-1051 (/: 27%, /srv: 25%, /var/lib/docker: 27%): RECOVERY disk space OK [18:17:43] !log Cleaned disk on integration-agent-docker-1051 [18:17:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:18:22] my guess is 44G is no more sufficient for /srv and running up to 4 builds in parallel [18:19:02] (03PS1) 10Subramanya Sastry: Add "2nd opinion needed?" column to diffs output [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1114441 [19:09:32] (03PS1) 10Ejegg: Test fundraising branches against php8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1114447 [19:14:32] hi releng, I'm trying to get the fundraising/REL1_43 branch passing tests, and I think I need something like ^^^ [19:14:47] Here's the patch that's failing now: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1113231 [19:16:52] 10Deployments, 06Release-Engineering-Team, 06SRE, 13Patch-For-Review: Reduce automatic messages on #wikimedia-operations - https://phabricator.wikimedia.org/T384804#10498487 (10Quiddity) Side-note in case it helps anyone (and probably only potentially helps IRCCloud users): I've been using some custom-CSS... [19:34:02] (03update) 10addshore: Updates on top of 0.26.0 (till 27th Jan) [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/600 (https://phabricator.wikimedia.org/T381646) [19:34:37] (03merge) 10addshore: Updates on top of 0.26.0 (till 27th Jan) [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/600 (https://phabricator.wikimedia.org/T381646) [20:14:58] (03update) 10dancy: spiderpig: CAS auth integration [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/625 [20:39:15] 10Deployments, 06Release-Engineering-Team, 06SRE, 13Patch-For-Review, 07SRE-Unowned: Reduce automatic messages on #wikimedia-operations - https://phabricator.wikimedia.org/T384804#10498758 (10RLazarus) [20:40:24] (03CR) 10Jforrester: "This is for DonationInterface with core, but do you also want to move DI itself, plus cldr and FundraisingEmailUnsubscribe here?" [integration/config] - 10https://gerrit.wikimedia.org/r/1114447 (owner: 10Ejegg) [21:28:49] (03update) 10dancy: spiderpig: CAS auth integration [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/625 [21:29:00] (03update) 10dancy: spiderpig: CAS auth integration [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/625 [21:52:12] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta cluster down: TypeError: Wikibase\Client\Hooks\LinkedDataSchemaGenerator::createSchemaElement(): Argument #5 ($description) must be of type string, null given, called in /srv/mediawiki/php-master/extensions/Wikibase/clien... - https://phabricator.wikimedia.org/T384877 [21:52:13] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta cluster down: TypeError: Wikibase\Client\Hooks\LinkedDataSchemaGenerator::createSchemaElement(): Argument #5 ($description) must be of type string, null given, called in /srv/mediawiki/php-master/extensions/Wikib... - https://phabricator.wikimedia.org/T384877#10499001 [21:53:43] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta cluster down: TypeError: Wikibase\Client\Hooks\LinkedDataSchemaGenerator::createSchemaElement(): Argument #5 ($description) must be of type string, null given, called in /srv/mediawiki/php-master/extensions/Wikib... - https://phabricator.wikimedia.org/T384877#10499005 [21:53:44] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T382365#10499006 (10matmarex) [21:54:17] (03close) 10egardner: Interaction: Move Interaction out of JobCard [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/639 [21:56:14] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta cluster down: TypeError: Wikibase\Client\Hooks\LinkedDataSchemaGenerator::createSchemaElement(): Argument #5 ($description) must be of type string, null given, called in /srv/mediawiki/php-master/extensions/Wikib... - https://phabricator.wikimedia.org/T384877#10499008 [21:59:21] (03approved) 10egardner: JobCard: prevent event bubbling and fix highlighted card [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/652 (https://phabricator.wikimedia.org/T383835) (owner: 10lwatson) [21:59:56] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta cluster down: TypeError: Wikibase\Client\Hooks\LinkedDataSchemaGenerator::createSchemaElement(): Argument #5 ($description) must be of type string, null given, called in /srv/mediawiki/php-master/extensions/Wikib... - https://phabricator.wikimedia.org/T384877#10499031 [22:06:58] (03merge) 10dancy: JobCard: prevent event bubbling and fix highlighted card [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/652 (https://phabricator.wikimedia.org/T383835) (owner: 10lwatson) [23:12:37] 10Beta-Cluster-Infrastructure, 10Wikidata, 10MW-1.44-notes (1.44.0-wmf.14; 2025-01-28): Beta cluster down: TypeError: Wikibase\Client\Hooks\LinkedDataSchemaGenerator::createSchemaElement(): Argument #5 ($description) must be of type string, null given, call... - https://phabricator.wikimedia.org/T384877#10499200 [23:22:50] (03CR) 10Arlolra: [C:03+2] Add "2nd opinion needed?" column to diffs output [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1114441 (owner: 10Subramanya Sastry) [23:35:38] (03CR) 10Arlolra: [C:03+2] Hide sitenotices & catlink boxes -- source of useless diff noise (032 comments) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1114109 (owner: 10Subramanya Sastry) [23:36:09] (03Merged) 10jenkins-bot: Hide sitenotices & catlink boxes -- source of useless diff noise [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1114109 (owner: 10Subramanya Sastry) [23:36:25] (03CR) 10Arlolra: Migrate cloudvps-configs/scripts to tools/ (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1114110 (owner: 10Subramanya Sastry)