[02:00:23] 06Project-Admins, 06Data Products, 06GLAM: Create project tag for  - https://phabricator.wikimedia.org/T369386#9963811 (10GFontenelle_WMF) Thank you so much for this, @Aklapper! [05:40:20] (03open) 10aklapper: Log creation when duplicating an EditEngine form [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/64 (https://phabricator.wikimedia.org/T369568) [08:13:09] 06Release-Engineering-Team, 06Java-Scala-Standardization, 10Data-Platform-SRE (2024.07.08 - 2024.07.28): Setup a test project to validate upload to the Gitlab package registry - https://phabricator.wikimedia.org/T367391#9964262 (10Gehel) [08:13:16] (03PS3) 10Arthur taylor: Add Wikibase client-only jobs to EntitySchema checks [integration/config] - 10https://gerrit.wikimedia.org/r/1052669 (https://phabricator.wikimedia.org/T367156) [08:13:25] 06Release-Engineering-Team, 06Java-Scala-Standardization, 10Data-Platform-SRE (2024.07.08 - 2024.07.28): Setup a test project to validate upload to the Gitlab package registry - https://phabricator.wikimedia.org/T367391#9964264 (10Gehel) a:03Gehel [08:13:44] (03CR) 10Arthur taylor: Add Wikibase client-only jobs to EntitySchema checks (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1052669 (https://phabricator.wikimedia.org/T367156) (owner: 10Arthur taylor) [08:18:44] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: Quibble CI images based on Buster fail to build due to sury.org dropping support - https://phabricator.wikimedia.org/T369146#9964283 (10hashar) [08:26:10] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team, 06collaboration-services, 06Data-Platform-SRE, 06Java-Scala-Standardization: Create a global Maven package registry in Gitlab - https://phabricator.wikimedia.org/T367322#9964317 (10Gehel) >>! In T367322#9948904, @LSobanski wrote: >... [08:26:58] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team, 06collaboration-services, 06Data-Platform-SRE, 06Java-Scala-Standardization: Create a global Maven package registry in Gitlab - https://phabricator.wikimedia.org/T367322#9964321 (10Gehel) >>! In T367322#9961605, @LSobanski wrote: >... [08:33:00] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team, 06collaboration-services, 06Data-Platform-SRE, 06Java-Scala-Standardization: Create a global Maven package registry in Gitlab - https://phabricator.wikimedia.org/T367322#9964324 (10Gehel) Current archiva is backed up. We should kee... [08:55:28] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team, 06collaboration-services, 06Data-Platform-SRE, 06Java-Scala-Standardization: Create a global Maven package registry in Gitlab - https://phabricator.wikimedia.org/T367322#9964414 (10LSobanski) So just to make sure it's 100% clear, w... [08:57:24] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.43.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T366958#9964435 (10Aklapper) 05Open→03In progress [08:59:47] hashar: I told you last week. Production images are running icu 67, not 63, and have for ages. [09:01:02] James_F: yeah [09:01:35] Hence the odd Wikimedia component of 'libicu67' pulling in a bespoke version of PHP 7.4. [09:01:56] Not my ideal in how packaging would go. :-) [09:02:07] 06Release-Engineering-Team, 10Data Pipelines, 06Data-Engineering, 10Data-Platform-SRE (2024.07.08 - 2024.07.28), 13Patch-For-Review: Upgrade Airflow to 2.9.2 - https://phabricator.wikimedia.org/T365449#9964473 (10Stevemunene) The airflow build process is currently stuck on two jobs namely `build-conda-en... [09:02:07] I'd like it to be validated and notably I'd like to know the upgrade plans as well as ensure the images are kept in sync [09:02:17] instead of having us catching up changes made to production [09:02:20] Yes, I would too. [09:02:26] hence the email [09:02:27] :) [09:02:56] and iirc, last week you told me tests are working with libicu67 [09:03:27] Yes, the tests work on both libicu67 (which is good, because prod runs it) and libicu63 (because CI enforces it). [09:03:29] sury.org has dropped support for Buster ( https://phabricator.wikimedia.org/T369146 ) so I guess we can go ahead and switch the non php74 jobs to bullseye [09:03:40] Yeah, I suppose we have to. [09:03:48] eventually ;] [09:03:57] Unless there's a Wikimedia PHP 8.1 we can switch to? [09:04:08] But I don't think so. I've been trying to make those images for them. [09:06:44] well prod is still on 7.4 [09:06:57] and I don't think we will upgrade to 8.1 anytime soon [09:10:05] Eh. I really hope we do. [09:14:35] oh for sure, or even straight to 8.3 [09:30:35] hashar: Should we switch the non-7.4 jobs over? [09:31:08] probably yeah [09:31:18] I mean [09:31:20] Give me a sec. [09:31:23] we have two set of images [09:31:28] Yeah. [09:31:48] 1) for production which should match whatever is in prod (my email) which is well Buster / php7.4 built by wmf [09:32:27] 2) for 3rd parties, which needs a set of php version and we can use whatever Debian version (though libicu, ffmpeg, chrome etc get updated as well which breaks thing) [09:32:43] there is that thing about Chromium now dieing out in headless mode when it is given arguments [09:32:50] Oh, right. [09:32:58] Yeah, we can't switch until that's fixed and back-ported. [09:33:03] cause it "can't accept multiple targets" [09:33:10] which is the most terrible message really [09:33:15] Yeah. [09:33:23] and added in an unrelated patch [09:33:24] anyway [09:34:16] and I haven't looked at what is injected to wdio cause somehow I 'd expect empty arguments to vanishes [09:35:30] $ command -V dumpargs [09:35:31] dumpargs () { python3 -c 'import pprint, sys; pprint.pprint(sys.argv[1:])' "${@}" } [09:35:31] $ dumpargs '' '' [09:35:31] ['', ''] [09:35:33] so really [09:35:45] after 30 years on the internet and more so dealing with computers [09:35:52] sometime I wonder whether I should stop here :D [09:36:00] :-D [09:36:31] but you pinned the probable root cause: wdio config / wdio-mediawiki that is erroneously passing empty arguments [09:36:33] leading to chrome failing [09:36:41] which causes CI to not be able to be switched to bullseye [09:36:56] which prevents images to be rebuild to upgrade Quibble to 1.9.2 [09:37:07] Quite possibly. [09:37:14] which blocks codders and kosta to make EntitySchema run phpunit tests in parrallel [09:37:16] and well [09:37:20] But wdio-mediawiki is something I've tried to avoid. [09:37:32] that is when my brain explode [09:38:07] 4D chess is really the bare minimum [09:40:53] I am catching up with emails & paperwork today [09:47:50] AH yeah https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1020314 [09:48:07] I guess I will review/test that this afternoon [09:48:10] + backports [09:48:32] Oh, I wrote that two months ago and forgot about it. [09:48:44] I don't remember if it fixes the problem. [09:49:45] hashar: can you help me find the job definition of the following ci pipeline? [09:49:45] https://integration.wikimedia.org/ci/job/wikidata-query-gui-build/ [09:49:45] I grepped in integration/config repo but could not find the specific job (only templates). I'd be interested in what commands are executed [09:53:45] jelto: https://gerrit.wikimedia.org/g/integration/config/+/refs/heads/master/jjb/wikidata.yaml#134 [09:54:05] the actual commands executed are a few lines above [09:54:19] they are shell scripts committed in the source repo [09:54:40] great thanks a lot :) [09:56:19] jelto: and the scripts are https://gerrit.wikimedia.org/g/wikidata/query-builder/+/refs/heads/master/scripts/ & https://gerrit.wikimedia.org/g/wikidata/query/gui/+/refs/heads/master/scripts/ [09:56:41] we created those ages ago [09:57:07] Noa Rave from WMDE recently revisited/polished them , they'd know more about it ;) [09:57:18] great thank you :) [09:57:35] ah the task was https://phabricator.wikimedia.org/T328543 [09:57:39] [QB] [WDQS-GUI] Move build scripts from CI to the repository [09:57:47] I am off, I gotta prepare lunch for kids :D [09:58:05] thanks for all the links, that unblocked me! [10:17:30] (03PS7) 10Arthur taylor: Add support for core's composer `phpunit:...parallel:...` commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/1039669 (https://phabricator.wikimedia.org/T365976) [10:33:54] (03CR) 10CI reject: [V:04-1] Add support for core's composer `phpunit:...parallel:...` commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/1039669 (https://phabricator.wikimedia.org/T365976) (owner: 10Arthur taylor) [10:58:26] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team, 06collaboration-services, 06Data-Platform-SRE, 06Java-Scala-Standardization: Create a global Maven package registry in Gitlab - https://phabricator.wikimedia.org/T367322#9964978 (10Gehel) >>! In T367322#9964414, @LSobanski wrote: >... [10:58:59] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team, 06collaboration-services, 06Java-Scala-Standardization, 10Data-Platform-SRE (2024.07.08 - 2024.07.28): Create a global Maven package registry in Gitlab - https://phabricator.wikimedia.org/T367322#9964983 (10Gehel) [10:59:39] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team, 06collaboration-services, 06Java-Scala-Standardization, 10Data-Platform-SRE (2024.07.08 - 2024.07.28): Create a global Maven package registry in Gitlab - https://phabricator.wikimedia.org/T367322#9964984 (10Gehel) a:03Gehel [11:11:41] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team, 06collaboration-services, 06Java-Scala-Standardization, 10Data-Platform-SRE (2024.07.08 - 2024.07.28): Create a global Maven package registry in Gitlab - https://phabricator.wikimedia.org/T367322#9965007 (10LSobanski) Backup steps... [11:46:47] 10Phabricator: Phabricator shows “it is a mystery” for some images - https://phabricator.wikimedia.org/T369621 (10LucasWerkmeister) 03NEW [12:18:32] 10Phabricator (Upstream), 07Upstream: No preview thumbnail for image file with more than 16,777,216 pixels - https://phabricator.wikimedia.org/T369621#9965181 (10Aklapper) p:05Triage→03Low [12:18:41] 10Phabricator (Upstream), 07Upstream: No preview thumbnail for image file with more than 16,777,216 pixels - https://phabricator.wikimedia.org/T369621#9965185 (10Aklapper) Per https://phabricator.wikimedia.org/file/transforms/56308614/ : > This image (with dimensions 5,750px x 3,281px) is too large to transfo... [12:58:19] 10Gerrit (Gerrit 3.10), 13Patch-For-Review: Use Gerrit 3.10 built-in log rotation - https://phabricator.wikimedia.org/T367505#9965266 (10hashar) I have confirmed the systemd timer is gone and no more rotating: ` sudo journalctl -u clear_gerrit_logs --since "2 days ago" -- Journal begins at Fri 2024-06-07 23:22... [13:01:47] 10Gerrit (Gerrit 3.10), 13Patch-For-Review: Use Gerrit 3.10 built-in log rotation - https://phabricator.wikimedia.org/T367505#9965279 (10hashar) And surely after restarting Gerrit on gerrit2002, the logs got cleaned: ` ls -lt /var/log/gerrit/|tail -n7 -r--r--r-- 1 gerrit2 gerrit2 1972 Jun 10 08:24 error_log... [13:03:23] 10Gerrit (Gerrit 3.10), 13Patch-For-Review: Use Gerrit 3.10 built-in log rotation - https://phabricator.wikimedia.org/T367505#9965281 (10hashar) a:03hashar [13:52:29] (03PS1) 10Giuseppe Lavagetto: zuul: remove conftool [integration/docroot] - 10https://gerrit.wikimedia.org/r/1052986 (https://phabricator.wikimedia.org/T369594) [14:44:50] (03CR) 10Hashar: [C:03+2] zuul: remove conftool [integration/docroot] - 10https://gerrit.wikimedia.org/r/1052986 (https://phabricator.wikimedia.org/T369594) (owner: 10Giuseppe Lavagetto) [14:45:30] (03Merged) 10jenkins-bot: zuul: remove conftool [integration/docroot] - 10https://gerrit.wikimedia.org/r/1052986 (https://phabricator.wikimedia.org/T369594) (owner: 10Giuseppe Lavagetto) [14:51:54] 10Gerrit (Gerrit 3.10), 13Patch-For-Review: Use Gerrit 3.10 built-in log rotation - https://phabricator.wikimedia.org/T367505#9965677 (10hashar) Same effect on gerrit1003: ` gerrit1003:~$ ls -lt /var/log/gerrit/|tail -n7 -r--r--r-- 1 gerrit2 gerrit2 410968 Jun 10 08:29 replication_log.2024-06-09.gz -r--r--r-... [14:53:26] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 06Data-Platform-SRE, 10Wikidata, and 3 others: move commons-query.wikimedia.org and query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#9965684 (10Jelto) [14:57:43] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 06Data-Platform-SRE, 10Wikidata, and 3 others: move commons-query.wikimedia.org and query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#9965711 (10Jelto) I imported the query-gui repo to GitLab: https://gitlab.wikim... [14:59:25] (03PS2) 10Hashar: Remove conftool job [integration/config] - 10https://gerrit.wikimedia.org/r/1052984 (https://phabricator.wikimedia.org/T369594) (owner: 10Giuseppe Lavagetto) [15:00:56] (03CR) 10Hashar: [C:03+2] "I have slightly amended to follow our convention:" [integration/config] - 10https://gerrit.wikimedia.org/r/1052984 (https://phabricator.wikimedia.org/T369594) (owner: 10Giuseppe Lavagetto) [15:02:04] (03Merged) 10jenkins-bot: Remove conftool job [integration/config] - 10https://gerrit.wikimedia.org/r/1052984 (https://phabricator.wikimedia.org/T369594) (owner: 10Giuseppe Lavagetto) [15:13:20] 10Beta-Cluster-Infrastructure, 06Machine-Learning-Team, 06Moderator-Tools-Team, 10ORES, 10PageTriage: Special:NewPagesFeed broken on beta cluster testwiki - https://phabricator.wikimedia.org/T349635#9965766 (10isarantopoulos) [15:29:53] 10Phabricator maintenance bot: Maintenance_bot not removing patch-for-review when all conditions are seemingly met - https://phabricator.wikimedia.org/T363573#9965807 (10Nemoralis) 05Resolved→03Open @Ladsgroup this is happening again (for different projects): T369342, T369485, T369319 [15:30:44] hashar: Nice find of text inside screenshots. But also :-((( [15:31:37] James_F: maybe the error was mentionned in raw text as well [15:31:58] the thing is that it is mentioned for ooui as well https://phabricator.wikimedia.org/T342302 [15:32:22] and well that got copy pastaed at various place https://codesearch.wmcloud.org/search/?q=CHROMIUM_FLAGS&files=&excludeFiles=&repos= [15:32:35] maybe Quibble injects too many spaces :/ [15:35:34] FIRING: Queue (Jenkins jobs + Zuul functions) alert: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [15:38:33] holy shit that’s a lot of CheckUser stuff in zuul ._. [15:40:03] hmm 2018-03-29 13:00:15 [15:40:06] a six years old bug [15:40:08] Lucas_WMDE: RIP CI [15:40:14] 10Continuous-Integration-Infrastructure, 10Quibble: Quibble jobs fail on bullseye images in QUnit job: "Multiple targets are not supported" - https://phabricator.wikimedia.org/T366799#9965881 (10hashar) It is also a bug in #quibble??! ` lang=python def chromium_flags(): args = [os.environ.get('CHROMIUM_FLA... [15:40:36] * Lucas_WMDE presses F [15:41:11] presses ALT + F4 [15:42:07] PROBLEM - Work requests waiting in Zuul Gearman server on contint1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [15:48:33] 10Phabricator maintenance bot: Maintenance_bot not removing patch-for-review when all conditions are seemingly met - https://phabricator.wikimedia.org/T363573#9965935 (10Ladsgroup) 05Open→03Resolved Fixed, the docker container went on zombie mode. New changes should get it fixed. I'll run it to backfill... [15:54:00] (03PS1) 10Itamar Givon: Update Query Builder test runners to node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1053016 [15:54:45] 10Phabricator, 13Patch-For-Review: Explore restricting editing task priority to Trusted-Contributors/staff/etc - https://phabricator.wikimedia.org/T363337#9966003 (10Aklapper) [15:55:34] RESOLVED: Queue (Jenkins jobs + Zuul functions) alert: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [15:57:33] (03PS1) 10Hashar: Fix "Multiple targets are not supported" with Chrome [integration/quibble] - 10https://gerrit.wikimedia.org/r/1053017 (https://phabricator.wikimedia.org/T366799) [15:57:46] https://gerrit.wikimedia.org/r/c/integration/quibble/+/1053017 Fix "Multiple targets are not supported" with Chrome [15:57:48] * hashar whistles [15:59:42] I am out for dinner preparation [16:00:52] (03CR) 10CI reject: [V:04-1] Fix "Multiple targets are not supported" with Chrome [integration/quibble] - 10https://gerrit.wikimedia.org/r/1053017 (https://phabricator.wikimedia.org/T366799) (owner: 10Hashar) [16:09:00] (03update) 10dduvall: ci: Try building directly from GitLab CI Git URL [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/113 [16:21:34] FIRING: Queue (Jenkins jobs + Zuul functions) alert: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [16:26:17] (03PS1) 10Pppery: Review access change [sandbox] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1053020 [16:26:36] (03CR) 10Pppery: [V:03+2 C:03+2] Review access change [sandbox] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1053020 (owner: 10Pppery) [16:27:30] (03Abandoned) 10Pppery: Review access change [sandbox] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1053020 (owner: 10Pppery) [16:41:34] RESOLVED: Queue (Jenkins jobs + Zuul functions) alert: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [17:00:13] RECOVERY - Work requests waiting in Zuul Gearman server on contint1002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [17:07:38] FIRING: Queue (Jenkins jobs + Zuul functions) alert: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [17:17:13] PROBLEM - Work requests waiting in Zuul Gearman server on contint1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [17:22:13] RECOVERY - Work requests waiting in Zuul Gearman server on contint1002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [17:22:38] RESOLVED: Queue (Jenkins jobs + Zuul functions) alert: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [20:16:59] (03open) 10dduvall: image: Support remote build and config contexts [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/101 [20:17:02] (03update) 10dduvall: image: Support remote build and config contexts [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/101 [20:37:29] (03update) 10dduvall: image: Support remote build and config contexts [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/101 [20:37:33] (03update) 10dduvall: image: Support remote build and config contexts [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/101 [20:37:38] (03update) 10dduvall: image: Support remote build and config contexts [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/101 [20:37:41] (03update) 10dduvall: image: Support remote build and config contexts [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/101 [20:49:21] (03PS2) 10Hashar: Fix "Multiple targets are not supported" with Chrome [integration/quibble] - 10https://gerrit.wikimedia.org/r/1053017 (https://phabricator.wikimedia.org/T366799) [21:06:22] (03CR) 10CI reject: [V:04-1] Fix "Multiple targets are not supported" with Chrome [integration/quibble] - 10https://gerrit.wikimedia.org/r/1053017 (https://phabricator.wikimedia.org/T366799) (owner: 10Hashar) [21:27:10] (03CR) 10Kosta Harlan: "add an entry to the changelog?" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1053017 (https://phabricator.wikimedia.org/T366799) (owner: 10Hashar) [22:59:38] 10Fresh, 07ARM support, 07Upstream: ECONNREFUSED error when running Selenium tests on M1 Mac - https://phabricator.wikimedia.org/T308889#9967634 (10Mhurd) I see that same error: ` nobody@bae9efe5b9a5:/test$ /usr/bin/chromium --version The hardware on this system lacks support for the sse3 instruction set. T...