[02:04:00] 10Gerrit: Gerrrit group ownership configuration oddities - https://phabricator.wikimedia.org/T327848 (10Pppery) [04:24:01] Gerrit seems to be taking 5-10 seconds to load diffs and change sets as of a few hours. \cc hashar [04:44:26] Project beta-scap-sync-world build #87638: 04FAILURE in 9 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/87638/ [04:50:34] Project beta-scap-sync-world build #87639: 04STILL FAILING in 2 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/87639/ [04:52:08] 10Beta-Cluster-Infrastructure: beta-scap-sync-world failure — 'No space left on device' - https://phabricator.wikimedia.org/T327853 (10TheresNoTime) [04:56:41] 10Beta-Cluster-Infrastructure: beta-scap-sync-world failure — 'No space left on device' - https://phabricator.wikimedia.org/T327853 (10TheresNoTime) ah, `mwmaint02` is at 100% usage `lang=s samtar@deployment-mwmaint02:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 2.0G 0 2.0G 0... [04:56:49] Project beta-scap-sync-world build #87640: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/87640/ [05:06:26] 10Beta-Cluster-Infrastructure: beta-scap-sync-world failure — 'No space left on device' - https://phabricator.wikimedia.org/T327853 (10TheresNoTime) `lang=s 9.2 GiB [##########] /srv 5.3 GiB [##### ] /var 2.6 GiB [## ] /usr 520.4 MiB [ ] /opt 51.0 MiB [ ] /boot 39... [05:10:52] Yippee, build fixed! [05:10:53] Project beta-scap-sync-world build #87641: 09FIXED in 4 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/87641/ [06:05:53] Krinkle: ack. The only thing I can see in the Grafana boards is we had a spike of load/cpu around 4:16 UTC which is a few minutes before your message [06:07:48] and we don't have the latency metrics for getting a diff or the change details (they are missing from https://grafana.wikimedia.org/d/IDkFbYeWz/latency-upstream? ) [06:13:54] beside not much more informations :-] [06:28:19] 10Beta-Cluster-Infrastructure: beta-scap-sync-world failure — 'No space left on device' - https://phabricator.wikimedia.org/T327853 (10hashar) Last week `deployment-jobrunner04` had its 10G `/srv` partition filed which we solved by resizing the OpenStack volume T327329. This `deployment-mwmaint02` has a similar... [06:46:28] 10Gerrit, 10Gerrit-Privilege-Requests: Gerrrit group ownership configuration oddities - https://phabricator.wikimedia.org/T327848 (10hashar) [09:20:08] 10Scap: scap deploy --init on deployment server fails on first puppet run - https://phabricator.wikimedia.org/T257317 (10hashar) More or less the same issue happened when adding a new repo yesterday ( https://gerrit.wikimedia.org/r/c/operations/puppet/+/860837 ). Puppet failed on `scap deploy --init`: ` $ sudo r... [11:18:44] Ah I figured out the udev rule to disable my webcam autofocus https://amusso.blogspot.com/2023/01/disable-webcam-autofocus-on-linux-when.html ;) [11:18:47] I am happy [12:42:08] 10Scap: scap deploy --init on deployment server fails on first puppet run - https://phabricator.wikimedia.org/T257317 (10hashar) Running the command manually: ` deploy1002:/srv/deployment/releng/jenkins-deploy$ scap deploy --init > /dev/null Incomplete setup: git_repo must be defined in the configuration ` So t... [12:45:27] hashar: ack, glad it's not something more mundane or serious. Will keep an eye out. [13:27:41] PROBLEM - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [13:28:49] PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [13:30:17] RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 67102 bytes in 0.043 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [13:30:47] RECOVERY - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is OK: OK - Certificate gerrit.wikimedia.org will expire on Wed 01 Mar 2023 09:47:05 PM GMT +0000. https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [13:44:39] * Krinkle is delighted to see hashar's blog show up in my feed reader! [13:50:25] Krinkle: link? [13:51:44] http://amusso.blogspot.com/ [13:53:52] thx [16:05:29] :] [16:05:55] it is seldomly updated. There are a few things I have blog about as a note keeper [16:40:20] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10larissagaulia) Hi, I'm the Engineering Manager for the performance team. Could I be added to Project-Admins so I can create tags, projects etc. I'm aware of the [[ https://www.mediaw... [16:55:08] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10bd808) >>! In T706#8558076, @larissagaulia wrote: > Hi, I'm the Engineering Manager for the performance team. Could I be added to Project-Admins so I can create tags, projects etc. I... [17:58:07] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: - https://phabricator.wikimedia.org/T327939 (10ntsako) [18:02:13] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: gdi - https://phabricator.wikimedia.org/T327939 (10Dzahn) [19:00:30] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: gdi - https://phabricator.wikimedia.org/T327939 (10Dzahn) @ntsako "access" is a bit of a vague term in this context. Since we are not creating a project but a namespace/group, it means that the owners/maintainers of this... [19:24:26] 10GitLab, 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱): Convert runner-1030.gitlab-runners.eqiad1.wikimedia.cloud to a shared runner - https://phabricator.wikimedia.org/T327949 (10dancy) [19:25:00] 10GitLab, 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱): Convert runner-1030.gitlab-runners.eqiad1.wikimedia.cloud to a shared runner - https://phabricator.wikimedia.org/T327949 (10dancy) [19:37:05] !log Taking runner-1030.gitlab-runners.eqiad1.wikimedia.cloud offline for T327949 [19:37:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:37:08] T327949: Convert runner-1030.gitlab-runners.eqiad1.wikimedia.cloud to a shared runner - https://phabricator.wikimedia.org/T327949 [19:53:27] 10GitLab, 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱): Convert runner-1030.gitlab-runners.eqiad1.wikimedia.cloud to a shared runner - https://phabricator.wikimedia.org/T327949 (10thcipriani) p:05Triage→03Medium [20:37:30] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: gdi - https://phabricator.wikimedia.org/T327939 (10Aklapper) [20:53:39] 10GitLab, 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱): Convert runner-1030.gitlab-runners.eqiad1.wikimedia.cloud to a shared runner - https://phabricator.wikimedia.org/T327949 (10dancy) [20:53:42] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab, 10User-brennen: Provision untrusted instance-wide GitLab job runners to handle user-level projects and merge requests from forks - https://phabricator.wikimedia.org/T297426 (10dancy) [21:35:55] !log runner-1030.gitlab-runners.eqiad1.wikimedia.cloud has been re-registered as an instance-wide runner. T327949 [21:35:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:35:57] T327949: Convert runner-1030.gitlab-runners.eqiad1.wikimedia.cloud to a shared runner - https://phabricator.wikimedia.org/T327949 [21:39:30] 10GitLab, 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱): Convert runner-1030.gitlab-runners.eqiad1.wikimedia.cloud to an instance-wide shared runner - https://phabricator.wikimedia.org/T327949 (10dancy) [21:40:55] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) >>! In T318521#8551236, @taavi wrote: >> @taavi are you able to re-create that setup for `gitlab-prod-1002` in `devtools` project as well (similar to `gitlab-prod-1001`... [21:50:01] (03PS1) 10Subramanya Sastry: Tweak FIXMEs to NOTEs since they aren't really fixmes. [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/883664 [21:50:22] (03CR) 10Subramanya Sastry: [C: 03+2] "Self-merging since this is trivial." [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/883664 (owner: 10Subramanya Sastry) [21:51:00] (03Merged) 10jenkins-bot: Tweak FIXMEs to NOTEs since they aren't really fixmes. [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/883664 (owner: 10Subramanya Sastry)