[00:05:35] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10Patch-For-Review: Scap3-ify Phabricator - https://phabricator.wikimedia.org/T313953 (10Dzahn) also see T313259#8234768 [00:06:42] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10Patch-For-Review, 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10Dzahn) [00:08:35] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10Patch-For-Review, 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10Dzahn) I edited the check boxes a bit (there is no phab1002, instead there is phab1004, phab2002 can be used for testing first, phab200... [05:40:45] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for TThoabala - https://phabricator.wikimedia.org/T317734 (10TThoabala) [10:07:52] Project beta-scap-sync-world build #68525: 04FAILURE in 3 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68525/ [10:17:09] Yippee, build fixed! [10:17:10] Project beta-scap-sync-world build #68526: 09FIXED in 2 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68526/ [10:55:41] Project beta-scap-sync-world build #68527: 15ABORTED in 30 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68527/ [10:56:52] hmmm what did you get stuck on.. [11:00:18] looks to be failing to `sync-apaches` [11:12:56] Project beta-scap-sync-world build #68528: 04FAILURE in 15 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68528/ [11:18:06] 10Beta-Cluster-Infrastructure: beta-scap-sync-world failing - https://phabricator.wikimedia.org/T317759 (10TheresNoTime) [11:18:10] Yippee, build fixed! [11:18:10] Project beta-scap-sync-world build #68529: 09FIXED in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68529/ [11:18:19] ... [11:24:23] 10Beta-Cluster-Infrastructure: beta-scap-sync-world failing - https://phabricator.wikimedia.org/T317759 (10TheresNoTime) And it's fixed itself... https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68529/console [11:25:06] 10Beta-Cluster-Infrastructure: beta-scap-sync-world failing - https://phabricator.wikimedia.org/T317759 (10TheresNoTime) 05Openβ†’03Invalid [13:36:41] Project beta-scap-sync-world build #68542: 04FAILURE in 7 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68542/ [13:43:29] Project beta-scap-sync-world build #68543: 04STILL FAILING in 4 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68543/ [13:44:28] 10Beta-Cluster-Infrastructure: beta-scap-sync-world failing - https://phabricator.wikimedia.org/T317759 (10TheresNoTime) 05Invalidβ†’03Open and it's back! [13:48:36] Project beta-scap-sync-world build #68544: 04STILL FAILING in 3 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68544/ [13:57:06] Yippee, build fixed! [13:57:07] Project beta-scap-sync-world build #68545: 09FIXED in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68545/ [14:43:07] TheresNoTime: It seems to be related to deployment-parsoid12.deployment-prep.eqiad1.wikimedia.clou [14:43:09] d [14:43:47] oh, I see you filed a ticket. [14:45:09] I've seen that a few times in the past. [14:47:23] 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Make scap deploy to kubernetes together with the legacy systems - https://phabricator.wikimedia.org/T299648 (10dancy) >>! In T299648#8234573, @dancy wrote: > @joe (and others) Is there any objection to scap running... [15:02:34] dancy: it seems very intermittent, so I personally wouldn't normally bother with a task, but it's caused `beta-scap-sync-world` to get stuck for ~30 minutes today [15:13:10] Nod. Maybe we can catch it in the act some time and debug it. [15:39:29] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10Patch-For-Review, 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10dduvall) [15:40:24] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10Patch-For-Review: Scap3-ify Phabricator - https://phabricator.wikimedia.org/T313953 (10dduvall) 05Openβ†’03Resolved a:03dduvall [16:06:43] (03CR) 10Hashar: "I talked about that new feature of scap to Dan before our team meeting. That surfaced after I have looked at the Phabricator checks script" [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 (owner: 10Hashar) [16:30:08] 10Continuous-Integration-Config, 10Release-Engineering-Team (Blocking 🧱), 10MediaWiki-Documentation, 10MW-1.40-notes (1.40.0-wmf.2; 2022-09-19), 10Patch-For-Review: MediaWiki core docs unavailable for MW 1.35 and later - https://phabricator.wikimedia.org/T317451 (10thcipriani) [16:31:51] 10Gerrit, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap, 10Patch-For-Review: Automate Gerrit deployment steps - https://phabricator.wikimedia.org/T317412 (10thcipriani) [16:33:01] (03PS1) 10Dduvall: checks: Avoid TypeError exception during check execution [tools/scap] - 10https://gerrit.wikimedia.org/r/832291 (https://phabricator.wikimedia.org/T313954) [16:33:07] 10Release-Engineering-Team (Radar), 10Scap: Catalog of errors from scap (or subprocesses) related to php-fpm restart always being enabled - https://phabricator.wikimedia.org/T311181 (10dancy) [16:38:06] 10Release-Engineering-Team (Seen), 10Scap: Make "scap backport" skip syncing steps for labs-only changes - https://phabricator.wikimedia.org/T317242 (10thcipriani) We should avoid leaving unsync'd changes in the `/srv/mediawiki-staging` directory. Is this task about syncing to all servers or is it about synci... [16:38:13] (03PS2) 10Dduvall: checks: Avoid TypeError exception during check execution [tools/scap] - 10https://gerrit.wikimedia.org/r/832291 (https://phabricator.wikimedia.org/T313954) [16:41:02] 10Release-Engineering-Team (Seen), 10Scap: Make "scap backport" skip syncing steps for labs-only changes - https://phabricator.wikimedia.org/T317242 (10TheresNoTime) Syncing is fairly quick β€” could this task not instead be about not doing the`restart php-fpm` step if all that's being targeted is the beta clust... [16:41:14] 10Release-Engineering-Team, 10Infrastructure-Foundations, 10SRE-tools: Investigate sharing releng common python code to pywmflib - https://phabricator.wikimedia.org/T316757 (10thcipriani) 05Openβ†’03Declined For the time being, I don't think we have anything appropriate to upstream. [16:43:16] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap: Strictly lock scap requirements dependencies - https://phabricator.wikimedia.org/T316755 (10thcipriani) [16:43:56] 10Beta-Cluster-Infrastructure, 10SRE-OnFire, 10Sustainability (Incident Followup): Add basic alerting to the Beta Cluster - https://phabricator.wikimedia.org/T315695 (10TheresNoTime) [16:44:19] 10Phabricator, 10Release-Engineering-Team (Radar), 10observability: Flapping Phabricator processes monitoring - https://phabricator.wikimedia.org/T315962 (10thcipriani) [16:45:19] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Radar), 10Upstream: Jobs started failing on https://releases-jenkins.wikimedia.org on 2022-08-21 - https://phabricator.wikimedia.org/T315897 (10thcipriani) 05Openβ†’03Stalled Awaiting upstream realease [16:46:08] 10Gerrit, 10Release-Engineering-Team (Seen): Enable a custom Gitiles header title - https://phabricator.wikimedia.org/T315888 (10thcipriani) p:05Triageβ†’03Low [16:46:39] 10Phabricator, 10Release-Engineering-Team (Blocking 🧱), 10User-AKlapper: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10thcipriani) [16:48:10] 10Release-Engineering-Team (Radar), 10Foundational Technology Requests: Deploy to Wikimedia Production from GitLab - https://phabricator.wikimedia.org/T315253 (10thcipriani) [16:52:20] 10Release-Engineering-Team (Seen), 10Tools: Migrate versions tool to noc.wikimedia.org - https://phabricator.wikimedia.org/T314897 (10thcipriani) [16:53:31] 10Continuous-Integration-Config, 10Metrics-Platform, 10Patch-For-Review: Standardize Java build for metrics-platform - https://phabricator.wikimedia.org/T314630 (10thcipriani) [16:53:51] 10Continuous-Integration-Config, 10Release-Engineering-Team (Radar), 10Metrics-Platform, 10Patch-For-Review: Standardize Java build for metrics-platform - https://phabricator.wikimedia.org/T314630 (10thcipriani) [16:54:15] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Radar): Stop triggering `beta-scap-sync-world` on `beta-mediawiki-config-update-eqiad` completion - https://phabricator.wikimedia.org/T314378 (10dancy) All done? [16:54:17] 10Release-Engineering-Team (Blocking 🧱), 10Data-Engineering-Radar, 10Gerrit-Privilege-Requests: Requesting membership of the analytics group in gerrit for 'snwachukwu' and 'nokafor' - https://phabricator.wikimedia.org/T314592 (10thcipriani) [16:54:47] 10GitLab (Administration, Settings & Policy), 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Create a form for tracking administrative privilege requests for GitLab - https://phabricator.wikimedia.org/T314495 (10thcipriani) [16:55:26] 10GitLab (Administration, Settings & Policy), 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10User-brennen: Create a form for tracking administrative privilege requests for GitLab - https://phabricator.wikimedia.org/T314495 (10thcipriani) [16:55:32] 10Release-Engineering-Team (Blocking 🧱), 10Projects-Cleanup: Archive or delete wikidata/query/flink-swift-plugin - https://phabricator.wikimedia.org/T314273 (10thcipriani) [16:56:53] 10Beta-Cluster-Infrastructure: beta-scap-sync-world failing - https://phabricator.wikimedia.org/T317759 (10TheresNoTime) My "monitoring" ([[ https://uptime.theresnotime.io/status/beta-cluster | this ]]) shows: {F35520524,size=full} for `instance-deployment-parsoid12` [16:57:46] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Radar): Stop triggering `beta-scap-sync-world` on `beta-mediawiki-config-update-eqiad` completion - https://phabricator.wikimedia.org/T314378 (10TheresNoTime) a:03taavi as far as I can tell, yup! :) [17:00:36] (03CR) 10Ahmon Dancy: [C: 03+2] "Looks reasonable to me." [tools/scap] - 10https://gerrit.wikimedia.org/r/832291 (https://phabricator.wikimedia.org/T313954) (owner: 10Dduvall) [17:01:37] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Radar): Stop triggering `beta-scap-sync-world` on `beta-mediawiki-config-update-eqiad` completion - https://phabricator.wikimedia.org/T314378 (10dancy) 05Openβ†’03Resolved [17:01:39] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Radar), 10Patch-For-Review, 10Upstream: Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597 (1... [17:05:35] 10Release-Engineering-Team (Seen), 10Scap: Make "scap backport" skip syncing steps for labs-only changes - https://phabricator.wikimedia.org/T317242 (10Zabe) >>! In T317242#8236809, @thcipriani wrote: > We should avoid unsync'd changes to `operations/mediawiki-config`. Is there a specific reason for that? Cha... [17:05:37] (03Merged) 10jenkins-bot: checks: Avoid TypeError exception during check execution [tools/scap] - 10https://gerrit.wikimedia.org/r/832291 (https://phabricator.wikimedia.org/T313954) (owner: 10Dduvall) [17:14:52] (03PS1) 10Dduvall: Release 4.19.1-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/832297 [17:23:51] 10Release-Engineering-Team, 10Infrastructure-Foundations, 10SRE-tools: Investigate sharing releng common python code to pywmflib - https://phabricator.wikimedia.org/T316757 (10Volans) Ok, no problem. Feel free to re-open if that changes. [17:44:28] (03CR) 10Dduvall: "I like the concept, and I think it could be a very valuable feature. The `before-` prefix feels a little clunky to me and I wonder if we c" [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 (owner: 10Hashar) [17:52:05] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.19.1-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/832297 (owner: 10Dduvall) [17:56:59] (03Merged) 10jenkins-bot: Release 4.19.1-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/832297 (owner: 10Dduvall) [18:34:43] 10Release-Engineering-Team, 10Scap: scap backport sanity check prevents using it to fix an about-to-be-live branch - https://phabricator.wikimedia.org/T317795 (10dancy) [18:36:41] 10Beta-Cluster-Infrastructure, 10SRE-OnFire, 10Sustainability (Incident Followup): Add basic alerting to the Beta Cluster - https://phabricator.wikimedia.org/T315695 (10TheresNoTime) a:03TheresNoTime [18:50:39] 10Phabricator: New Herald rule: add `design-systems-team` tag to any new tasks tagged with `codex` - https://phabricator.wikimedia.org/T317801 (10ldelench_wmf) [19:20:10] 10Release-Engineering-Team, 10Scap: scap backport sanity check prevents using it to fix an about-to-be-live branch - https://phabricator.wikimedia.org/T317795 (10jeena) what about having it do everything except sync if it's not deployed anywhere? [19:23:08] (03PS1) 10Ahmon Dancy: Perform k8s deployments in parallel [tools/scap] - 10https://gerrit.wikimedia.org/r/832338 (https://phabricator.wikimedia.org/T299648) [19:23:26] 10Beta-Cluster-Infrastructure: prometheus-beta.wmcloud.org 504 timeout - https://phabricator.wikimedia.org/T315699 (10TheresNoTime) 05Openβ†’03Resolved a:03TheresNoTime If it's not DNS, it's the firewall /s (it was the firewall.) [19:29:14] 10Beta-Cluster-Infrastructure: prometheus-beta.wmcloud.org 504 timeout - https://phabricator.wikimedia.org/T315699 (10TheresNoTime) 05Resolvedβ†’03Open It occurs to me that perhaps this //shouldn't// be publicly accessible β€” rather than leave it up and wait for a reply, I've undone my change (and confirmed htt... [20:04:44] (03PS1) 10Ahmon Dancy: fixed a typo [tools/scap] - 10https://gerrit.wikimedia.org/r/832341 [20:05:17] (03CR) 10Ahmon Dancy: [C: 03+2] fixed a typo [tools/scap] - 10https://gerrit.wikimedia.org/r/832341 (owner: 10Ahmon Dancy) [20:08:35] (03Merged) 10jenkins-bot: fixed a typo [tools/scap] - 10https://gerrit.wikimedia.org/r/832341 (owner: 10Ahmon Dancy) [20:26:17] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for TThoabala - https://phabricator.wikimedia.org/T317734 (10thcipriani) Hi @TThoabala thanks for your interest in deployment training! You should have an invite to the UTC Early training session on Sept 15thβ€”let me know if yo... [20:26:32] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for TThoabala - https://phabricator.wikimedia.org/T317734 (10thcipriani) p:05Triageβ†’03Medium [20:27:37] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for TheresNoTime - https://phabricator.wikimedia.org/T305191 (10thcipriani) 05Openβ†’03Resolved a:03thcipriani Calling this one resolved! [20:27:49] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for jnuche - https://phabricator.wikimedia.org/T310191 (10thcipriani) 05Openβ†’03Resolved a:03thcipriani [20:29:41] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for TheresNoTime - https://phabricator.wikimedia.org/T305191 (10thcipriani) a:05thciprianiβ†’03ArielGlenn [20:29:52] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for jnuche - https://phabricator.wikimedia.org/T310191 (10thcipriani) a:05thciprianiβ†’03ArielGlenn [20:39:16] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10Jelto) Some updates from troubleshooting the networking issue: I see two independent problems here: *... [20:43:04] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Tool-spacemedia, 10Toolforge: Grant access to tool-spacemedia gitlab repository - https://phabricator.wikimedia.org/T317805 (10Don-vip) [20:46:50] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10thcipriani) >>! In T308271#8237437, @Jelto wrote: > I think Trusted Runner need outgoing internet acces... [20:52:12] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap: scap backport sanity check prevents using it to fix an about-to-be-live branch - https://phabricator.wikimedia.org/T317795 (10thcipriani) [20:56:09] 10Release-Engineering-Team, 10Infrastructure-Foundations, 10SRE-tools: Investigate sharing releng common python code to pywmflib - https://phabricator.wikimedia.org/T316757 (10thcipriani) >>! In T316757#8236998, @Volans wrote: > Ok, no problem. Feel free to re-open if that changes. Thank you for answering a... [21:02:49] 10Release-Engineering-Team (Blocking 🧱), 10Data Pipelines (Sprint 01): Increase maximum artifacts size on Gitlab Registry to accommodate files >1GB - https://phabricator.wikimedia.org/T317555 (10Antoine_Quhen) Sorry, your change didn't make it work. But I can now confirm that the limit is server-side. In thi... [21:03:02] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Tool-spacemedia: Grant access to tool-spacemedia gitlab repository - https://phabricator.wikimedia.org/T317805 (10JJMC89) [21:07:43] (03PS1) 10Ahmon Dancy: devel hacks: Change SCAP to REMOTE_SCAP [tools/scap] - 10https://gerrit.wikimedia.org/r/832353 [21:25:55] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Tool-spacemedia: Grant access to tool-spacemedia gitlab repository - https://phabricator.wikimedia.org/T317805 (10Dzahn) @Don-vip I just added you to https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/project_members as owner. [21:28:16] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Tool-spacemedia: Grant access to tool-spacemedia gitlab repository - https://phabricator.wikimedia.org/T317805 (10Dzahn) About your other point about notifications: There had been an announcement here https://lists.wikimedia.org/hyperkitty/list/cl... [21:28:47] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Tool-spacemedia: Grant access to tool-spacemedia gitlab repository - https://phabricator.wikimedia.org/T317805 (10Dzahn) @Don-vip Does it work for you now? [21:29:01] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Tool-spacemedia: Grant access to tool-spacemedia gitlab repository - https://phabricator.wikimedia.org/T317805 (10Dzahn) a:03Don-vip [21:37:45] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Tool-spacemedia: Grant access to tool-spacemedia gitlab repository - https://phabricator.wikimedia.org/T317805 (10Don-vip) 05Openβ†’03Resolved @Dzahn yes it works, thank you! :) [21:44:46] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: wm-juniors-il - https://phabricator.wikimedia.org/T313750 (10Dzahn) a:03Dzahn [21:44:56] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: wm-juniors-il - https://phabricator.wikimedia.org/T313750 (10Dzahn) 05Openβ†’03In progress [21:51:37] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Projects-Cleanup, 10User-dduvall: Create Blubber repo on GitLab, archive Gerrit repo - https://phabricator.wikimedia.org/T307533 (10Dzahn) [21:54:18] 10Gerrit, 10Projects-Cleanup, 10Release Pipeline (Blubber): Archive the Blubber gerrit repo - https://phabricator.wikimedia.org/T317820 (10Dzahn) [21:54:36] 10Gerrit, 10Projects-Cleanup, 10Release Pipeline (Blubber): Archive the Blubber gerrit repo - https://phabricator.wikimedia.org/T317820 (10Dzahn) [21:54:38] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Projects-Cleanup, 10User-dduvall: Create Blubber repo on GitLab, archive Gerrit repo - https://phabricator.wikimedia.org/T307533 (10Dzahn) [21:55:49] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Projects-Cleanup, 10User-dduvall: Create Blubber repo on GitLab, archive Gerrit repo - https://phabricator.wikimedia.org/T307533 (10Dzahn) created subtask via template as requested by Hashar [22:01:59] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: wm-juniors-il - https://phabricator.wikimedia.org/T313750 (10Dzahn) Hi @Matanya I was wondering if we should create a tree structure for Wikimedia Israel, /repos/wm Project: repos / Wikimedia Israel / Wikijuniors (/re... [22:29:02] (03PS1) 10Ahmon Dancy: ProgressReporter: Prefix progress indicator with a timestamp [tools/scap] - 10https://gerrit.wikimedia.org/r/832373 [22:31:01] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10dduvall) I second that: Thanks for digging in :) >>! In T308271#8237437, @Jelto wrote: > I tried to co... [22:31:58] (03PS7) 10Jeena Huneidi: Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) [22:32:19] (03CR) 10Jeena Huneidi: Replace Lock with TimeoutLock (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [22:33:13] (03PS8) 10Jeena Huneidi: Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) [22:38:51] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10Patch-For-Review, 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10dduvall) Here's the result of a recent test deploy to phab2002: ` [...] returned [1]: Executing check 'finalize' Check 'finalize' fail... [22:45:54] (03PS1) 10Dduvall: dockerfiles: Create and chown home directory for buildctl user [integration/config] - 10https://gerrit.wikimedia.org/r/832374 (https://phabricator.wikimedia.org/T308271) [22:47:24] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10Patch-For-Review, 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10Dzahn) >>! In T313259#8237860, @dduvall wrote: > I verified manually that this is indeed a timeout to `m3-slave.codfw.wmnet` and not in... [22:56:29] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10Patch-For-Review, 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10Dzahn) It seems like we are not supposed to use default port 3306 but one of 3321 or 3322 (judging by the iptables rules on db2160). No... [23:17:24] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10serviceops, 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) [23:17:30] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10Patch-For-Review, 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10Dzahn) [23:17:37] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10serviceops, 10serviceops-collab, 10Patch-For-Review: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) 05Resolvedβ†’03Open [23:17:43] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10serviceops, 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) [23:20:40] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10Patch-For-Review, 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10Dzahn) reopening T315713 because I could confirm that this works: `mysql -h m3-slave.eqiad.wmnet -P 3323 -u phstats -D phabricator_pro... [23:32:03] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10serviceops, 10serviceops-collab, 10Patch-For-Review: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) @Marostegui I am reopening this because I think we need to ask for grant changes after a... [23:42:01] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10Patch-For-Review, 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10Dzahn) So `modules/profile/templates/mariadb/grants/production-m3.sql.erb` has 259 lines and basically all of it are phab related GRANT... [23:48:20] 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10serviceops, 10serviceops-collab, 10Patch-For-Review: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) The short form is we need what is in `modules/profile/templates/mariadb/grants/productio... [23:53:08] 10Phabricator, 10Release-Engineering-Team (Radar), 10observability, 10Patch-For-Review: Flapping Phabricator processes monitoring - https://phabricator.wikimedia.org/T315962 (10Dzahn) >>! In T315962#8180510, @fgiunchedi wrote: > I agree 100% with you here, please consider ditching the `check_procs` bit and... [23:53:17] 10Phabricator, 10Release-Engineering-Team (Radar), 10observability, 10Patch-For-Review: Flapping Phabricator processes monitoring - https://phabricator.wikimedia.org/T315962 (10Dzahn) 05Openβ†’03Resolved [23:58:13] 10GitLab (Project Migration), 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10Epic, and 2 others: Migrate active repositories in Phabricator Differential to GitLab - https://phabricator.wikimedia.org/T191182 (10Dzahn) a:03Dzahn [23:59:23] 10GitLab (Project Migration), 10Phabricator, 10Release-Engineering-Team (Bonus Level πŸ•ΉοΈ), 10Epic, and 2 others: Migrate active repositories in Phabricator Differential to GitLab - https://phabricator.wikimedia.org/T191182 (10Dzahn) I have done parts of this in T296022 already.