[00:01:43] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:06:43] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:07:42] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for haak - https://phabricator.wikimedia.org/T357312 (10brennen) 05Open→03Resolved [00:13:14] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for haak - https://phabricator.wikimedia.org/T357312 (10bd808) This user's `cn` led to finding and fixing {T357328} [00:22:27] 10GitLab (Account Approval), 10Release-Engineering-Team, 10User-brennen: Requesting GitLab account activation for Cyberwolf - https://phabricator.wikimedia.org/T355692 (10brennen) a:03brennen I don't see a `cyberwolfdev` account on the GitLab instance currently - could you double-check your username? [00:41:51] 10Release-Engineering-Team (Seen), 10Wikimedia Design Style Guide: Use `git lfs` for large binary files of Design Style Guide - https://phabricator.wikimedia.org/T235013 (10Volker_E) [00:42:50] 10Release-Engineering-Team (Seen), 10Wikimedia Design Style Guide: Use `git lfs` for large binary files of Design Style Guide - https://phabricator.wikimedia.org/T235013 (10Volker_E) 05Open→03Declined This is ready to be declined with #Codex and it's Codex Figma files now in place. [02:55:07] 10GitLab (Account Approval), 10Release-Engineering-Team, 10User-brennen: Requesting GitLab account activation for Cyberwolf - https://phabricator.wikimedia.org/T355692 (10thcipriani) 05Open→03Resolved Seems like the account approver bot got this one already: https://wikitech.wikimedia.org/wiki/Tool:Gitla... [06:08:24] 10Continuous-Integration-Infrastructure, 10EntitySchema, 10Wikidata, 10Wikidata Dev Team, 10wmde-wikidata-tech: [ES] Investigate OutOfMemory error in browser test, remove workaround - https://phabricator.wikimedia.org/T356402 (10hashar) > Fatal error: Allowed memory size of 134217728 bytes exhausted...... [06:32:56] 10Continuous-Integration-Infrastructure, 10EntitySchema, 10Wikidata, 10Wikidata Dev Team, 10wmde-wikidata-tech: [ES] Investigate OutOfMemory error in browser test, remove workaround - https://phabricator.wikimedia.org/T356402 (10hashar) `$wgLocalisationCacheConf['store'] = 'array';` that is the format us... [06:37:31] 10Continuous-Integration-Infrastructure, 10LibUp, 10ci-test-error: Git repos with direct pushes bypassing gerrit/CI could be a maintenance burden for libup patches - https://phabricator.wikimedia.org/T301201 (10hashar) Thank you @Umherirrender ! [06:38:38] 10Continuous-Integration-Infrastructure, 10LibUp, 10ci-test-error: Git repos with direct pushes bypassing gerrit/CI could be a maintenance burden for libup patches - https://phabricator.wikimedia.org/T301201 (10hashar) [06:50:17] 10Gerrit, 10Release-Engineering-Team (Seen): Expand Gerrit Manager permissions - https://phabricator.wikimedia.org/T234474 (10hashar) [09:34:29] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.8), 10Patch-For-Review: Upgrade to Gerrit 3.8 - https://phabricator.wikimedia.org/T354886 (10hashar) > {icon exclamation-triangle color=yellow} **[[ https://gerrit-review.googlesource.com/360219 | Change 360219]]: Delete vote now fails for an already deleted vot... [09:35:49] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.8), 10Patch-For-Review: Upgrade to Gerrit 3.8 - https://phabricator.wikimedia.org/T354886 (10hashar) [10:31:43] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10MediaWiki-Core-Tests, 10Code-Health, and 8 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10kostajh) >>! In T225730#9535536, @Daimona wrote: > Slow CI can, at tim... [10:43:49] (03PS1) 10Hashar: Let Gerrit manage light/dark theme [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/1002931 (https://phabricator.wikimedia.org/T354886) [10:43:55] (03PS1) 10Hashar: Support Gerrit 3.8 CSS styling API [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/1002932 (https://phabricator.wikimedia.org/T354886) [10:44:19] (03CR) 10CI reject: [V: 04-1] Support Gerrit 3.8 CSS styling API [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/1002932 (https://phabricator.wikimedia.org/T354886) (owner: 10Hashar) [10:46:38] (03PS2) 10Hashar: Support Gerrit 3.8 CSS styling API [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/1002932 (https://phabricator.wikimedia.org/T354886) [10:58:21] Hello :) As we move closer to 50% of external traffic on k8s, I wanted to check with y'all if scap checked errors coming from mw-on-k8s canaries when doing mediawiki deployments (no sense in me writing a ticket asking for it if it already does) [10:59:48] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.8), 10Patch-For-Review: Upgrade to Gerrit 3.8 - https://phabricator.wikimedia.org/T354886 (10hashar) [11:12:44] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, 10serviceops: Create parsoid mediawiki deployment - https://phabricator.wikimedia.org/T357392 (10akosiaris) [11:14:12] why some people have real names in gitlab but some don't? It's very hard to find people without names there. Compare https://gitlab.wikimedia.org/abi with https://gitlab.wikimedia.org/nikerabbit [11:17:08] random guess without knowing the gitlab setup, But i suspect its the ldap data https://ldap.toolforge.org/user/abi https://ldap.toolforge.org/user/nikerabbit [11:20:19] that's what gitlab says, but I haven't found a reason for the difference nor place to update [11:28:24] Nikerabbit: because our LDAP has been broken since we have created it :) [11:29:51] claime: jnuche might now about how scap checks the mw-on-k8s canaries [11:30:00] s/now/know [11:31:17] hashar: any fix in sight? I've heard of bitu or something [11:32:27] Nikerabbit: I think Gitlab authenticate the user against bitu yeah, and bitu uses the LDAP `cn` as the username [11:32:56] that is consistent with Gerrit which uses the `cn` as well [11:33:20] 10Continuous-Integration-Infrastructure, 10EntitySchema, 10Wikidata, 10Wikidata Dev Team, 10wmde-wikidata-tech: [ES] Investigate OutOfMemory error in browser test, remove workaround - https://phabricator.wikimedia.org/T356402 (10Krinkle) >>! In T356402#9536675, @hashar wrote: > `$wgLocalisationCacheConf[... [11:33:28] and that is the MediaWiki username on Wikitech (which is from where the `cn` came from originally) [11:34:04] so unless we get a name policy (we don't) and find a way to rename account (historically we have seen it is too complicated), that will keep being inconsistent [11:34:31] okay :/ [11:34:39] but theorically we could have a policy that WMF employees should have an account (and cn) whic uses first letter of firstname + last name [11:34:57] potentially with a WMF suffix [11:35:07] that is how office it creates our wiki accounts [11:35:16] or our @wikimedia.org email addresses [11:35:36] By now I have spreadsheet of all team members with links to phab/gerrit/github/gitlab to check their contributions. Just wondering how to make it easier for others. [11:35:45] but in practice, the techonboarding tells you to create whatever developer accounts and people pick random schemes [11:35:51] yes [11:36:15] that used to more or less work ten + years ago when one could more or less keep in mind the nickname/real name of the few dozen of developers [11:36:21] but that definitely does not work anymore [11:36:24] so in practice [11:36:36] instead of coming as `hashar` with the email `hashar@free.fr` [11:37:07] i guess I should have a new account such as `Antoine Musso (WMF)` or `amusso` with email `amusso@wikimedia.org` [11:40:39] Nikerabbit: the doc to create an account does suggest to use your full name ( https://wikitech.wikimedia.org/wiki/Help:Create_a_Wikimedia_developer_account#Create_a_Wikimedia_developer_account ) [11:40:47] and it is not changeable [11:41:16] so yeah it is complicated :-] [11:46:49] 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, and 2 others: Investigate container scanning options within the context of the Gitlab appsec pipeline - https://phabricator.wikimedia.org/T307523 (10acooper) Regarding this ticket, I would recommen... [12:03:32] claime: hi there, by looking at the code it looks like we are not checking the K8s canaries yet AFAICT, so creating that task makes sense methinks [12:04:03] jnuche: Ok, doing so then, thanks for checking [12:29:15] 10Release-Engineering-Team, 10Scap, 10MW-on-K8s, 10SRE, 10serviceops: Scap should check errors coming from mw-on-k8s canaries during deployments - https://phabricator.wikimedia.org/T357402 (10Clement_Goubert) [12:45:56] (03CR) 10Paladox: [C: 03+1] Let Gerrit manage light/dark theme [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/1002931 (https://phabricator.wikimedia.org/T354886) (owner: 10Hashar) [12:47:00] (03CR) 10Paladox: [C: 03+1] Support Gerrit 3.8 CSS styling API [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/1002932 (https://phabricator.wikimedia.org/T354886) (owner: 10Hashar) [13:27:00] (03CR) 10Hashar: [C: 03+2] Let Gerrit manage light/dark theme [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/1002931 (https://phabricator.wikimedia.org/T354886) (owner: 10Hashar) [13:27:34] (03Merged) 10jenkins-bot: Let Gerrit manage light/dark theme [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/1002931 (https://phabricator.wikimedia.org/T354886) (owner: 10Hashar) [13:31:58] well that one works [13:32:37] (03CR) 10Hashar: [C: 03+2] Support Gerrit 3.8 CSS styling API [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/1002932 (https://phabricator.wikimedia.org/T354886) (owner: 10Hashar) [13:33:10] (03Merged) 10jenkins-bot: Support Gerrit 3.8 CSS styling API [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/1002932 (https://phabricator.wikimedia.org/T354886) (owner: 10Hashar) [13:38:30] (03CR) 10Hashar: [C: 03+2] "I have tested it and it works! :)" [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/1002932 (https://phabricator.wikimedia.org/T354886) (owner: 10Hashar) [13:40:33] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.8), 10Patch-For-Review: Upgrade to Gerrit 3.8 - https://phabricator.wikimedia.org/T354886 (10hashar) [13:42:29] (03CR) 10Hashar: [C: 03+2] Gerrit 3.8 no more set redundant real_author [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/999928 (https://phabricator.wikimedia.org/T354886) (owner: 10Hashar) [13:42:47] (03PS3) 10Hashar: wm-checks-api: Gerrit 3.8 no more sets redundant real_author [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/999928 (https://phabricator.wikimedia.org/T354886) [13:43:23] (03CR) 10Hashar: [C: 03+2] wm-checks-api: Gerrit 3.8 no more sets redundant real_author [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/999928 (https://phabricator.wikimedia.org/T354886) (owner: 10Hashar) [13:44:06] (03Merged) 10jenkins-bot: wm-checks-api: Gerrit 3.8 no more sets redundant real_author [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/999928 (https://phabricator.wikimedia.org/T354886) (owner: 10Hashar) [13:46:12] hashar: where can I change the way the link to Zuul status page is made? [13:46:26] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/941003 -> https://integration.wikimedia.org/zuul/#q=941003,3 [13:46:30] ah [13:46:32] the ",3" is making the query not useful [13:46:37] in the Gerrit `Checks` tab isn't it? [13:46:38] it searches for 3 everywhere [13:46:42] hehe [13:46:48] Yeah, it's the in-progress link before the results are done [13:47:00] the red bubble with the external link icon [13:47:25] https://gerrit.wikimedia.org/r/operations/software/gerrit.git and branch `deploy/wmf/stable-3.7` [13:47:52] then it is somewhere in the javascript plugin which processes the Zuul live status.json: plugins/wm-zuul-status.js [13:49:22] const checkRun = { [13:49:23] statusLink: `https://integration.wikimedia.org/zuul/#q=${status.id}`, [13:50:56] and my guess the id being `` comes directly from Zuul [13:51:16] cause that is the unit of work Zuul knows about [13:52:04] I guess I never tested it : [13:52:04] D [13:55:22] oh [13:55:45] current_filter.toLowerCase().split( /[\s,]+/ ) [13:56:19] so yeah that is the Zuul status page splitting on `,` [13:56:48] Krinkle: maybe the Zuul filter input should only split on space? [13:57:13] I;d rather send only the change number. [13:57:23] This also has the benefit of making the link useful even after an amendment or rebase [13:57:43] ah true [13:57:54] the jobs stop/restart anyway [13:58:08] but then one might not reclick the link [13:58:22] and would thus be stall with a status page filtering on the previous and obsolete patchset [13:58:24] Yeah, that seems like a feature to not have to re-click the link just to get logically the same results. [13:58:44] There is no concept of outdated results afaik since there can't be jobs for a non-current patchset. [13:59:10] so I guess use: ${status.id.split(/,/)[0]} [14:00:04] Zuul has support to keep the old patchset in the queue [14:00:24] that is dequeue-on-patchset, but that is not the default [14:01:21] so it makses sense to only point to the change [14:01:40] I don't think the method is covered by a unit test, then I am not sure that is needed :) [14:05:15] Showing other results in case they are still running, seems also a feature, if someone else used this plugin, and they configured Zuul in that way, they presumably will receive those results eventually and continue to find them relevant. E.g. a potential variation of 'check experimental' might be run only on some patch set versions and not all. Either way, the results are relevant I think! [14:05:30] so this is somethign I would patch in operations/software/gerrit? in which branch should new patches start? [15:01:01] Krinkle: sorry I have missed your replies [15:01:44] repo: https://gerrit.wikimedia.org/r/operations/software/gerrit.git and branch `deploy/wmf/stable-3.7` [15:01:52] stable-3.7 is the upstream one [15:02:04] wmf/stable-3.7 is our fork in case we wanna cherry pick [15:02:30] but really the important one is the `deploy/wmf/stable-3.7` which has the artifacts to run Gerrit + the plugins [15:02:37] it is the default HEAD [15:03:00] so I think if you'd clone it you would end up in the proper branch [15:03:11] possibly use `--depth=1` or `--single-branch` [15:24:10] 10Beta-Cluster-Infrastructure, 10MediaWiki-Platform-Team: Set up some beta cluster wikis with different registrable domain - https://phabricator.wikimedia.org/T355281 (10pmiazga) I would stick to `beta.wmcloud.org` as it nicely sticks follows `{LANG}.{PROJECT}.org` schema. Now I wonder, do we need an additiona... [15:35:01] 10Release-Engineering-Team (Seen), 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 2 others: Create parsoid mediawiki deployment - https://phabricator.wikimedia.org/T357392 (10Jdforrester-WMF) [15:41:58] 10Continuous-Integration-Infrastructure, 10EntitySchema, 10Wikidata, 10wmde-wikidata-tech: [ES] Investigate OutOfMemory error in browser test, remove workaround - https://phabricator.wikimedia.org/T356402 (10karapayneWMDE) [15:59:19] 10Beta-Cluster-Infrastructure, 10MediaWiki-Platform-Team: Set up some beta cluster wikis with different registrable domain - https://phabricator.wikimedia.org/T355281 (10ArielGlenn) We may want to test the behaviour when going from logged in on a wiki on beta.wmflabs.org (let's say en.wikipedia) and then visit... [17:10:49] 10GitLab (Upstream pit of despair 🕳️), 10commit-message-validator: Add support for commit message trailers that GitLab markdown will render on individual lines - https://phabricator.wikimedia.org/T351253 (10bd808) [17:40:44] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, 10Patch-For-Review, and 2 others: Many (all?) of the phabricator/tools scripts are in Python 2 - https://phabricator.wikimedia.org/T355574 (10CodeReviewBot) dzahn merged https://gitlab.wikimedia.org/repos/phabricator/... [18:29:29] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, 10Patch-For-Review, and 2 others: Many (all?) of the phabricator/tools scripts are in Python 2 - https://phabricator.wikimedia.org/T355574 (10Dzahn) I resolved merge conflicts and merged this in gitlab. Then manually... [18:45:19] 10Project-Admins: Create project tag for Wikimedia Foundation Memory Bank project - https://phabricator.wikimedia.org/T357451 (10Varnent) [18:45:26] 10Phabricator (phabricator-next), 10User-bd808: Default Calendar shared search for "Month View" has nonsense saved value for "Tags" (#wmse-findingglams-2018-glam-events) - https://phabricator.wikimedia.org/T357014 (10brennen) [18:47:15] 10Phabricator (phabricator-next), 10Dumps-Generation, 10collaboration-services, 10Patch-For-Review, 10User-brennen: phabricator_task_dump.service Failed on phab1004 - https://phabricator.wikimedia.org/T355502 (10brennen) [18:48:15] 10Project-Admins, 10WMF-Communications: Create project tag for Wikimedia Foundation Memory Bank project - https://phabricator.wikimedia.org/T357451 (10Varnent) [18:50:09] 10Project-Admins, 10WMF-Communications: Create project tag for Wikimedia Foundation Memory Bank project - https://phabricator.wikimedia.org/T357451 (10Varnent) [19:32:32] 10Continuous-Integration-Config, 10I18n, 10Language-Team (Language-2024-January-March), 10Language-Technical Support (Language-Technical Support (Current) ), 10affects-translatewiki.net: Automatically allow
in message translations - https://phabricator.wikimedia.org/T356548 (10Winston_Sung) 05Open... [19:34:54] Sury has finally updated their package for php-excimer, so anyone using this in mediawiki-docker can now also use the Speedscope output locally (which TimStarling added in one of the later releases that is now included). https://github.com/oerdnj/deb.sury.org/issues/1952 [19:38:53] 10Release-Engineering-Team, 10Wikimedia-Extension-setup, 10Russian-Sites, 10Wikimedia-extension-review-queue: Add Extension:PlaceNewSection to Russian Wikipedia - https://phabricator.wikimedia.org/T344501 (10Iniquity) 05Stalled→03Open [19:39:49] 10Release-Engineering-Team, 10Release Pipeline (Blubber): Publish Blubber reference doc - https://phabricator.wikimedia.org/T356902 (10dduvall) 05Open→03Resolved a:03dduvall We recently did some work on Blubber (and other pipeline tooling) documentation in {T352259}, and part of the work entailed refacto... [20:07:10] 10Phabricator (phabricator-next), 10User-brennen: Deploy Phabricator/Phorge week of 2024-02-12 - https://phabricator.wikimedia.org/T357464 (10brennen) [20:10:17] 10Phabricator (phabricator-next), 10Patch-For-Review, 10User-brennen: Deploy Phabricator/Phorge week of 2024-02-12 - https://phabricator.wikimedia.org/T357464 (10CodeReviewBot) brennen opened https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/34 update tools & phabricator submodules... [20:15:21] 10Phabricator (phabricator-next), 10Patch-For-Review, 10User-brennen: Deploy Phabricator/Phorge week of 2024-02-12 - https://phabricator.wikimedia.org/T357464 (10CodeReviewBot) dzahn merged https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/34 update tools & phabricator submodules fo... [20:25:51] 10Phabricator (phabricator-next), 10Dumps-Generation, 10collaboration-services, 10Patch-For-Review, 10User-brennen: phabricator_task_dump.service Failed on phab1004 - https://phabricator.wikimedia.org/T355502 (10Dzahn) after T357464 I could now run the dump script: ` [phab1004:/srv/phab/tools] $ sudo /... [20:27:58] 10Phabricator (phabricator-next), 10Patch-For-Review, 10User-brennen: Deploy Phabricator/Phorge week of 2024-02-12 - https://phabricator.wikimedia.org/T357464 (10brennen) 05Open→03Resolved [20:33:39] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments, 10User-brennen: 1.42.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T354436 (10brennen) Stable on group0. Some sync timeouts encountered during `scap train-presync` logged at P56710. [20:47:09] 10Phabricator (2024-02-13), 10Dumps-Generation, 10collaboration-services, 10Patch-For-Review, 10User-brennen: phabricator_task_dump.service Failed on phab1004 - https://phabricator.wikimedia.org/T355502 (10Dzahn) 05Open→03In progress [20:47:12] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, 10Patch-For-Review, and 2 others: Many (all?) of the phabricator/tools scripts are in Python 2 - https://phabricator.wikimedia.org/T355574 (10Dzahn) [20:58:03] 10Release-Engineering-Team, 10collaboration-services: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355776 (10Dzahn) [20:58:05] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 10collaboration-services: GitLab email confirmation mail ends up in spam folder - https://phabricator.wikimedia.org/T346607 (10Dzahn) [20:59:11] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 10collaboration-services: GitLab email confirmation mail ends up in spam folder - https://phabricator.wikimedia.org/T346607 (10Dzahn) We should probably get back to this as kind of part of T355776. [21:05:05] raise ValueError('ZIP does not support timestamps before 1980') [21:05:08] of course not... [21:05:50] that is all because Bill Gates was born ten year too late [21:06:14] or he would have used 1970 for MSDOS like the rest of the worl^H^H^H^Hunixes [21:28:09] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: beta.wmflabs.org: Job infrastructure is down - https://phabricator.wikimedia.org/T357476 (10Urbanecm_WMF) [21:28:26] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: beta.wmflabs.org: Job infrastructure is down - https://phabricator.wikimedia.org/T357476 (10Urbanecm_WMF) Before the investigation, I checked at mwlog02 for the JobExecutor file – it did not exist. I started to investigate the issue, eventually notic... [21:37:47] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: beta.wmflabs.org: Job infrastructure is down - https://phabricator.wikimedia.org/T357476 (10Urbanecm_WMF) Ha...as I submitted this, I saw some `notificationKeepGoingJob` jobs in the `JobExecutor.log`. So maybe the jobrunners are now processing the bac... [21:50:03] !log add kostajh to acl*phabricator per T355405 [21:50:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:50:06] T355405: Add kostajh to acl*phabricator - https://phabricator.wikimedia.org/T355405 [21:50:31] 10Phabricator, 10Release-Engineering-Team: Add kostajh to acl*phabricator - https://phabricator.wikimedia.org/T355405 (10brennen) 05Open→03Resolved a:03brennen Done - sorry for the delay here, just noticed this. [21:52:45] 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team (Seen): Hide issues tab from GitLab navigation - https://phabricator.wikimedia.org/T357265 (10brennen) [21:53:09] 10Release-Engineering-Team (Radar), 10collaboration-services: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355776 (10brennen) [21:55:27] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Revert to using upstream src/infrastructure/daemon/workers/management/PhabricatorWorkerManagementWorkflow.php - https://phabricator.wikimedia.org/T352721 (10brennen) [21:56:35] 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team (Seen): Gerrit has some helpful diff views that it would be nice to have in Gitlab - https://phabricator.wikimedia.org/T349394 (10brennen) [21:57:23] 10GitLab (Auth & Access), 10Release-Engineering-Team (Radar), 10CAS-SSO, 10Infrastructure-Foundations, and 2 others: Add GitLab to offboarding workflow - https://phabricator.wikimedia.org/T339843 (10brennen) [22:00:05] 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team (Seen), 10Upstream: Cannot log into gitlab.wikimedia.org with LDAP username which has only one character - https://phabricator.wikimedia.org/T329728 (10brennen) [22:05:00] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: cg-data - https://phabricator.wikimedia.org/T357355 (10brennen) 05Open→03Resolved a:03brennen Created: https://gitlab.wikimedia.org/repos/cg-data [22:06:51] 10Phabricator (Upstream), 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10collaboration-services, and 2 others: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355691 (10brennen) [22:21:59] 10Phabricator, 10Release-Engineering-Team, 10collaboration-services, 10User-brennen: Allow tools to use phabricator webhooks - https://phabricator.wikimedia.org/T321790 (10Dzahn) I can talk to en.wikipedia from the phabricator prod machine (without setting a proxy): ` phab1004:~] $ curl -s https://en.wik... [22:35:01] 10Phabricator, 10Release-Engineering-Team, 10collaboration-services, 10User-brennen: Allow tools to use phabricator webhooks - https://phabricator.wikimedia.org/T321790 (10Dzahn) https://secure.phabricator.com/T6755 and https://secure.phabricator.com/D12136 are relevant reading material [23:12:26] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: beta.wmflabs.org: Job infrastructure is down - https://phabricator.wikimedia.org/T357476 (10Urbanecm_WMF) Okay. It seems `deployment-docker-cpjobqueue01` and `deployment-changeprop-1` were fighting with each other over who will run. Since `deployment-... [23:17:38] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 10collaboration-services: GitLab email confirmation mail ends up in spam folder - https://phabricator.wikimedia.org/T346607 (10Dzahn) > I tried resending confirmation instructions via https://gitlab.wikimedia.org/users/confirmation/new but didn't... [23:35:58] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 10collaboration-services: GitLab email confirmation mail ends up in spam folder - https://phabricator.wikimedia.org/T346607 (10Dzahn) The subject of the mail this ticket is about originally is "Confirmation instructions". When searching my own in... [23:52:24] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 10collaboration-services: GitLab email confirmation mail ends up in spam folder - https://phabricator.wikimedia.org/T346607 (10brennen) I confirmed @Dzahn's check of the current workflow. I do not think this is an issue any more, although I'm fuz... [23:54:04] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 10collaboration-services: GitLab email confirmation mail ends up in spam folder - https://phabricator.wikimedia.org/T346607 (10Dzahn) a:03Dzahn