[01:50:52] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for feeglgeef - https://phabricator.wikimedia.org/T386667 (10Feeglgeef) 03NEW [03:33:54] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for jgafnea - https://phabricator.wikimedia.org/T386501#10558265 (10Pppery) Welcome to Wikimedia, jgafnea! The bot isn't approving this because you don't meet any of the criteria to be bot-approved, which require som... [07:00:39] 10Phabricator-Bot-Requests: Phabricator Bot request for WMDE GitHub notifications - https://phabricator.wikimedia.org/T386382#10558398 (10Aklapper) > Would it be possible to change the email afterward? I do not think so, however it's not used for anything anyway. [07:38:22] 10Phabricator (Upstream), 07Upstream: Provide thumbnails on the file list - https://phabricator.wikimedia.org/T135729#10558460 (10Aklapper) First step would likely make File Transforms by default also create a "Thumbgrid (100px)"/`TRANSFORM_THUMBGRID`. Currently the code only creates a "Preview (220px)"/`TRANS... [09:19:17] 10Phabricator-Bot-Requests: Phabricator Bot request for WMDE GitHub notifications - https://phabricator.wikimedia.org/T386382#10558735 (10seanleong-WMDE) Okay! That's okay, I have changed the description with the email. Thanks! [09:20:48] 10Phabricator-Bot-Requests: Phabricator Bot request for WMDE GitHub notifications - https://phabricator.wikimedia.org/T386382#10558737 (10seanleong-WMDE) [10:09:42] 10Phabricator-Bot-Requests, 10Release-Engineering-Team (Priority Backlog šŸ“„): Phabricator Bot request for WMDE GitHub notifications - https://phabricator.wikimedia.org/T386382#10558888 (10Aklapper) 05Openā†’03Resolved a:03Aklapper @seanleong-WMDE Bot account created. You can find the access token in P73... [10:24:27] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 2 others: [SPIKE][Infra] Re-start Wikibase test coverage reporting - https://phabricator.wikimedia.org/T288396#10558935 (10ArthurTaylor) a:03ArthurTaylor [11:00:42] 10Phabricator-Bot-Requests, 10Release-Engineering-Team (Priority Backlog šŸ“„): Phabricator Bot request for WMDE GitHub notifications - https://phabricator.wikimedia.org/T386382#10559046 (10seanleong-WMDE) Got it, thanks! @Aklapper [12:05:36] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana-next.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [12:13:58] 10Phabricator, 07Performance Issue: Herald rules causing delays to task edit saves - getting worse - https://phabricator.wikimedia.org/T108586#10559168 (10Aklapper) [12:13:59] 10Phabricator: Disable personal Herald rule H334 - https://phabricator.wikimedia.org/T386458#10559169 (10Aklapper) [12:14:00] 10Phabricator: Disable personal Herald rules H154? - https://phabricator.wikimedia.org/T374800#10559171 (10Aklapper) [12:14:03] 10Phabricator: Disable personal Herald rule H448 - https://phabricator.wikimedia.org/T386457#10559170 (10Aklapper) [12:14:09] 10Phabricator: Disable personal Herald rules H124, H213? - https://phabricator.wikimedia.org/T371219#10559172 (10Aklapper) [12:14:17] 10Phabricator, 10Release-Engineering-Team (Priority Backlog šŸ“„): Herald rules H295 and H301 broken as their conditions list custom "date" fields somehow not supported by Herald - https://phabricator.wikimedia.org/T367830#10559173 (10Aklapper) [12:14:23] 10Phabricator: Disable Herald rule H99 (adding #Internet-Archive) - https://phabricator.wikimedia.org/T367650#10559174 (10Aklapper) [12:14:34] 10Phabricator: Disable personal Herald rule H77? - https://phabricator.wikimedia.org/T366855#10559175 (10Aklapper) [12:14:40] 10Phabricator: Disable personal Herald rule H37? - https://phabricator.wikimedia.org/T366854#10559176 (10Aklapper) [12:14:46] 10Phabricator: Disable Personal Herald rules H66, H236, H313 as the user currently cannot access Phab - https://phabricator.wikimedia.org/T248662#10559179 (10Aklapper) [12:14:52] 10Phabricator: Disable Masumrezarock100's Herald rule H335 - https://phabricator.wikimedia.org/T362059#10559177 (10Aklapper) [12:14:58] 10Phabricator, 14Release-Engineering-Team (Quid Pro Crow šŸ¦ƒ): Disable Differential related Herald rules - https://phabricator.wikimedia.org/T351585#10559178 (10Aklapper) [12:15:04] 10Phabricator, 07Performance Issue: Herald rules causing delays to task edit saves - getting worse - https://phabricator.wikimedia.org/T108586#10559181 (10Aklapper) On a related note, I proposed https://we.phorge.it/D25878 in upstream to check affected Herald rule conditions at project archival. [12:20:08] 10Phabricator (Upstream), 07Upstream: Allow admins to rename personal Herald rules - https://phabricator.wikimedia.org/T386703 (10Aklapper) 03NEW [12:25:36] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana-next.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [12:26:16] 10Phabricator Antivandalism Extension, 10Release-Engineering-Team (Doing šŸ˜Ž): More AVA performance improvements - https://phabricator.wikimedia.org/T386704 (10Aklapper) 03NEW p:05Triageā†’03Medium [12:30:56] Project mediawiki-core-doxygen build #8307: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/8307/ [12:45:32] (03PS1) 10Arthur taylor: Fix `suite.xml` coverage configuration generation for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/1120532 (https://phabricator.wikimedia.org/T288396) [12:54:04] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 3 others: [SPIKE][Infra] Re-start Wikibase test coverage reporting - https://phabricator.wikimedia.org/T288396#10559302 (10ArthurTaylor) Had another look at this. By now, the W... [13:16:39] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana-next.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [13:30:22] Yippee, build fixed! [13:30:22] Project mediawiki-core-doxygen build #8308: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/8308/ [13:36:39] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana-next.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [13:44:38] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana-next.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [13:59:38] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana-next.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [14:04:31] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana-next.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [14:09:46] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana-next.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [14:15:37] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana-next.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [14:35:38] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana-next.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [14:47:03] (03PS1) 10Ssingh: Revert "Archive operations/debs/dnsdist" [integration/config] - 10https://gerrit.wikimedia.org/r/1120566 [14:47:11] (03CR) 10CI reject: [V:04-1] Revert "Archive operations/debs/dnsdist" [integration/config] - 10https://gerrit.wikimedia.org/r/1120566 (owner: 10Ssingh) [15:10:58] hit an odd error when running pre-merge tests - anything to worry about or just try again? https://integration.wikimedia.org/ci/job/citoid-pipeline-rehearse/118/console [15:18:04] I donā€™t remember seeing that error before [15:18:09] found one occurrence in Phabricator, https://phabricator.wikimedia.org/T310643#8124340 [15:18:19] I would try again [15:31:02] yeah seeing it on a recheck [16:11:41] !log Zuul: [operations/debs/dnsdist] Revert archival [16:11:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:33:53] Could one of you fine folks in this channel who is a Gerrit admin help with T385602? [16:33:53] T385602: Request membership in extension-MathSearch group for HamidRahkooy - https://phabricator.wikimedia.org/T385602 [16:43:43] bd808: I'll process it. [16:43:52] thanks dancy [16:46:31] Seems like wikibugs may died [16:47:01] I just restarted it (per #wikimedia-operations) [16:47:08] ah, thanks Lucas [16:47:23] (03Merged) 10jenkins-bot: Zuul: Add Hamid Rahkooy to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1119815 (https://phabricator.wikimedia.org/T385602) (owner: 10Physikerwelt) [16:47:42] Gosh, welcome back wikibugs. [16:47:51] (Thanks Lucas_WMDE.) [16:49:52] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1119815 [16:49:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:56:51] 06Gerrit-Privilege-Requests, 10MathSearch, 13Patch-For-Review: Request membership in extension-MathSearch group for HamidRahkooy - https://phabricator.wikimedia.org/T385602#10560254 (10dancy) 05Openā†’03Resolved a:03dancy I added Hamid Rahkooy to the `extension-MathSearch` Gerrit group. [17:27:46] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for jgafnea - https://phabricator.wikimedia.org/T386501#10560379 (10jgafnea) Thanks, that's very helpful. Is there a general-ish rule about what code lives in Gerrit vs GitLab? [17:46:45] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for jgafnea - https://phabricator.wikimedia.org/T386501#10560433 (10Pppery) Very roughly: - Large complicated projects like MediaWiki and extensions tend to live on Gerrit. - Small self-contained projects are more li... [18:04:51] (03open) 10jnuche: allow `ci-charts` to publish a base image for the MW chart [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/109 (https://phabricator.wikimedia.org/T386702) [18:07:38] (03update) 10jnuche: allow `ci-charts` to publish a base image for the MW chart [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/109 (https://phabricator.wikimedia.org/T386702) [18:29:10] (03PS1) 10QChris: Allow ā€œGerrit Managersā€ to import history [labs/tools/wdaudiolex-be] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1120627 [18:29:10] (03CR) 10QChris: [V:03+2 C:03+2] Allow ā€œGerrit Managersā€ to import history [labs/tools/wdaudiolex-be] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1120627 (owner: 10QChris) [18:29:38] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/wdaudiolex-be] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1120629 [18:29:38] (03CR) 10QChris: [V:03+2 C:03+2] Import done. Revoke import grants [labs/tools/wdaudiolex-be] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1120629 (owner: 10QChris) [18:50:47] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team, 10Data-Engineering (Q3 2024 January 1st - March 31th): Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10560742 (10Ahoelzl) [18:50:54] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team, 06Data-Engineering: Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10560744 (10Ahoelzl) [19:09:56] Amir1: up from here [19:10:07] https://grafana.wikimedia.org/d/35WSHOjVk/application-servers-red-k8s?orgId=1&refresh=1m&viewPanel=17 [19:10:27] I meant to click -see [19:10:31] -sre [19:10:34] Here was above [19:16:19] dduvall: Any idea what makes pipelinelib go boom like ? The `git ls-remote https://gerrit.wikimedia.org/r/integration/pipelinelib` command that is blowing up seems to work from my local machine. [19:22:36] bd808: not sure but it hasn't loaded pipelinelib at that point [19:22:45] is it intermittent? [19:24:19] happened thrice in a row :( [19:26:21] That one is interesting/annoying. [19:33:20] does seem rather annoying and lacking any useful debugging info [20:30:44] dduvall: hmmm... I guess I am just now understanding that the failure happens trying to load pipelinelib itself. Hungry bd808 did a bad job of reading the error log. [20:33:20] Looking at the latest failure `Cannot run program "git": error=0, Failed to exec spawn helper: pid: 1261505, exit value: 1` is the crash cause and I think that must be happening on contint2002 itself? [20:57:43] Created T386755 to track the issue. Not quite sure how to get Jenkins to actually tell us anything useful about the `git` crash. [20:57:43] T386755: striker-pipeline-test failing to load pipelinelib with git error - https://phabricator.wikimedia.org/T386755 [21:15:08] 10Continuous-Integration-Infrastructure, 10Striker, 10ci-test-error (WMF-deployed Build Failure): striker-pipeline-test failing to load pipelinelib with git error - https://phabricator.wikimedia.org/T386755#10561327 (10bd808) [21:25:02] 10Phabricator, 07SecTeam-Processed: Change the dropdown in security ticket dropdown to not include WMF Product and WMF Technology as two separate departments - https://phabricator.wikimedia.org/T384243#10561362 (10sbassett) >>! In T384243#10556385, @Aklapper wrote: > How/for what is the security task author af... [21:39:32] thanks, bd808 [21:42:03] dduvall: if there is any rabbit hole I could go down to help figure out the pipelinelib bootstrapping problem in Striker just point me in a useful direction [21:46:12] 06Release-Engineering-Team, 10Scap, 06serviceops: OSError "Message too long" from scap helmfile diffs - https://phabricator.wikimedia.org/T386759 (10RLazarus) 03NEW [21:46:13] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for feeglgeef - https://phabricator.wikimedia.org/T386667#10561427 (10Hogue) I trust Feeglgeef and Feeglgeef wrote a Python script to check results of an analysis I made based on the Wikifunctionsdump. Now I am waitin... [21:48:13] i'm sure there are a number of jenkins related rabbit holes to go down. i'll poke around the pipeline shared library parts