[00:19:26] 10Continuous-Integration-Config, 10Pywikibot, 10Pywikibot-tests: Jenkins output for pywikibot job is hard to read - https://phabricator.wikimedia.org/T117570 (10Pppery) Can this be closed as resolved given that the patch was merged? [00:30:19] RECOVERY - Check systemd state on doc1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [01:13:04] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Phabricator deployment 2023-03-30 - https://phabricator.wikimedia.org/T333516 (10brennen) [01:14:13] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Phabricator deployment 2023-03-30 - https://phabricator.wikimedia.org/T333516 (10brennen) p:05Triage→03Medium [01:31:41] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review, 10User-brennen: Phabricator deployment 2023-03-30 - https://phabricator.wikimedia.org/T333516 (10Dzahn) [01:36:41] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, 10Patch-For-Review: contint2002 service implementation tracking - https://phabricator.wikimedia.org/T324659 (10Dzahn) The production role for ci::master is now applied on contint2002. Some minor follow-ups were needed: - run puppet mult... [01:37:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10serviceops-collab: contint hardware refresh - https://phabricator.wikimedia.org/T294276 (10Dzahn) [01:37:23] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, 10Patch-For-Review: contint2002 service implementation tracking - https://phabricator.wikimedia.org/T324659 (10Dzahn) 05Open→03In progress [02:07:38] 10Continuous-Integration-Config, 10Pywikibot, 10Pywikibot-tests: Jenkins output for pywikibot job is hard to read - https://phabricator.wikimedia.org/T117570 (10Xqt) This is still open because flake8-colours was merged to flake8 but does not work with the CI output any longer. [08:06:33] 10Continuous-Integration-Config, 10Pywikibot, 10Pywikibot-tests: Jenkins output for pywikibot job is hard to read - https://phabricator.wikimedia.org/T117570 (10hashar) In a lot of case the tools check whether the standard input is a TTY (`sys.stdint.isatty()`) and disable color output when it is not. On CI... [08:07:05] 10Continuous-Integration-Config, 10Pywikibot, 10Pywikibot-tests: Jenkins output for pywikibot job is hard to read - https://phabricator.wikimedia.org/T117570 (10hashar) Short of figuring the issue in `flake8`, the command line can be passed `--color=always` and that should work on CI. One can then confirm lo... [08:07:28] 10Phabricator, 10serviceops-collab: Phabricator's access log may have some problems in log rotation - https://phabricator.wikimedia.org/T332869 (10Jelto) Logrotate on `phab1004` looks good. The job run UTC night, exited successfully and alert resolved. ` Mar 30 00:00:55 phab1004 systemd[1]: logrotate.service:... [09:35:31] (03CR) 10Jforrester: jjb: Provide bespoke PHPUnit standalone job for MediaWiki core (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/904100 (https://phabricator.wikimedia.org/T203694) (owner: 10Jforrester) [09:37:15] (03CR) 10Jaime Nuche: "Is there value in updating the failing gerrit-git-fat-pull job to test lfs? From what I gather maybe it's not worth it we can just remove " [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/904239 (https://phabricator.wikimedia.org/T333465) (owner: 10Hashar) [09:43:35] 10Continuous-Integration-Config, 10Growth-Team, 10Moderator-Tools-Team, 10PageTriage: Add PageTriage to gated extensions - https://phabricator.wikimedia.org/T333534 (10kostajh) [09:45:15] (03PS1) 10Kosta Harlan: zuul: Add PageTriage to gatedextensions [integration/config] - 10https://gerrit.wikimedia.org/r/904488 (https://phabricator.wikimedia.org/T333534) [09:46:21] (03CR) 10CI reject: [V: 04-1] zuul: Add PageTriage to gatedextensions [integration/config] - 10https://gerrit.wikimedia.org/r/904488 (https://phabricator.wikimedia.org/T333534) (owner: 10Kosta Harlan) [09:46:58] 10Continuous-Integration-Config, 10Growth-Team, 10Moderator-Tools-Team, 10PageTriage, 10Patch-For-Review: Add PageTriage to gated extensions - https://phabricator.wikimedia.org/T333534 (10kostajh) a:03kostajh [09:47:46] 10Continuous-Integration-Config, 10Growth-Team, 10Moderator-Tools-Team, 10PageTriage, 10Patch-For-Review: Add PageTriage to gated extensions - https://phabricator.wikimedia.org/T333534 (10kostajh) [09:53:28] (03PS2) 10Kosta Harlan: zuul: Add PageTriage to gatedextensions [integration/config] - 10https://gerrit.wikimedia.org/r/904488 (https://phabricator.wikimedia.org/T333534) [09:53:47] 10Continuous-Integration-Config, 10Moderator-Tools-Team, 10PageTriage, 10Growth-Team (Current Sprint), 10Patch-For-Review: Add PageTriage to gated extensions - https://phabricator.wikimedia.org/T333534 (10kostajh) [09:55:32] (03CR) 10Kosta Harlan: [C: 04-1] "We need to resolve T333535 first." [integration/config] - 10https://gerrit.wikimedia.org/r/904488 (https://phabricator.wikimedia.org/T333534) (owner: 10Kosta Harlan) [10:17:50] (03PS3) 10Kosta Harlan: zuul: Add PageTriage to gatedextensions [integration/config] - 10https://gerrit.wikimedia.org/r/904488 (https://phabricator.wikimedia.org/T333534) [10:17:52] (03CR) 10Kosta Harlan: zuul: Add PageTriage to gatedextensions (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/904488 (https://phabricator.wikimedia.org/T333534) (owner: 10Kosta Harlan) [10:22:33] 10Phabricator, 10Release-Engineering-Team (Kanban), 10UX-Debt: make 'hidden' fields actually hidden on the phabricator form preview view - https://phabricator.wikimedia.org/T209743 (10valerio.bozzolan) This was now merged in Phorge itself (the fork of Phabricator): https://we.phorge.it/T15217 https://we.ph... [10:46:39] 10Continuous-Integration-Config, 10MediaWiki-extensions-CentralAuth: Add CentralAuth to gated extensions - https://phabricator.wikimedia.org/T333541 (10kostajh) [10:46:53] 10Continuous-Integration-Config, 10MediaWiki-extensions-CentralAuth: Add CentralAuth to gated extensions - https://phabricator.wikimedia.org/T333541 (10kostajh) a:03kostajh [10:47:12] (03PS1) 10Kosta Harlan: zuul: Add CentralAuth to gatedextensions [integration/config] - 10https://gerrit.wikimedia.org/r/904497 (https://phabricator.wikimedia.org/T333541) [10:52:29] (03CR) 10Hashar: jjb: Provide bespoke PHPUnit standalone job for MediaWiki core (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/904100 (https://phabricator.wikimedia.org/T203694) (owner: 10Jforrester) [12:03:36] 10Continuous-Integration-Config, 10MediaWiki-extensions-CentralAuth, 10Patch-For-Review: Add CentralAuth to gated extensions - https://phabricator.wikimedia.org/T333541 (10taavi) I've asked about this before, and my understanding is that the presence of CA completely breaks most of core's authentication tests. [12:44:29] (03CR) 10Jforrester: [C: 04-2] "Per task. :-(" [integration/config] - 10https://gerrit.wikimedia.org/r/904497 (https://phabricator.wikimedia.org/T333541) (owner: 10Kosta Harlan) [12:45:13] 10Continuous-Integration-Config, 10MediaWiki-extensions-CentralAuth, 10Patch-For-Review: Add CentralAuth to gated extensions - https://phabricator.wikimedia.org/T333541 (10Jdforrester-WMF) Yeah, this was previously declined sadly for that reason. :-( In the magical post-GitLab world I suppose we could build... [12:45:48] 10Continuous-Integration-Config, 10MediaWiki-extensions-CentralAuth, 10Patch-For-Review: Add CentralAuth to gated extensions - https://phabricator.wikimedia.org/T333541 (10Jdforrester-WMF) [12:45:50] 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10Epic, 10Patch-For-Review: Have all Wikimedia production extensions and skins in the CI gate - https://phabricator.wikimedia.org/T249674 (10Jdforrester-WMF) [12:45:57] 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10Epic, 10Patch-For-Review: Have all Wikimedia production extensions and skins in the CI gate - https://phabricator.wikimedia.org/T249674 (10Jdforrester-WMF) [12:49:40] 10Continuous-Integration-Config, 10MediaWiki-extensions-CentralAuth, 10Patch-For-Review: Add CentralAuth to gated extensions - https://phabricator.wikimedia.org/T333541 (10hashar) >>! In T333541#8741475, @taavi wrote: > I've asked about this before, and my understanding is that the presence of CA completely... [14:39:04] (03CR) 10Hashar: Migrate from git fat to git lfs (031 comment) [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/904239 (https://phabricator.wikimedia.org/T333465) (owner: 10Hashar) [14:43:09] (03PS1) 10Hashar: Remove gerrit-git-fat-pull job [integration/config] - 10https://gerrit.wikimedia.org/r/904551 (https://phabricator.wikimedia.org/T333465) [14:43:44] (03CR) 10Hashar: [C: 03+2] Remove gerrit-git-fat-pull job [integration/config] - 10https://gerrit.wikimedia.org/r/904551 (https://phabricator.wikimedia.org/T333465) (owner: 10Hashar) [14:44:45] (03Merged) 10jenkins-bot: Remove gerrit-git-fat-pull job [integration/config] - 10https://gerrit.wikimedia.org/r/904551 (https://phabricator.wikimedia.org/T333465) (owner: 10Hashar) [14:49:08] 10Phabricator, 10serviceops-collab, 10Patch-For-Review: Phabricator's access log may have some problems in log rotation - https://phabricator.wikimedia.org/T332869 (10eoghan) I forgot to detail it in the ticket, but I manually started the logrotate timer unit on `phab1004`. It seems as if we need to `systemc... [14:53:26] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review, 10User-brennen: Phabricator deployment 2023-03-30 - https://phabricator.wikimedia.org/T333516 (10brennen) [14:57:08] (03PS2) 10Hashar: Migrate from git fat to git lfs [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/904239 (https://phabricator.wikimedia.org/T333465) [14:58:01] (03CR) 10Hashar: Migrate from git fat to git lfs (031 comment) [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/904239 (https://phabricator.wikimedia.org/T333465) (owner: 10Hashar) [14:58:39] (03CR) 10Hashar: [C: 03+2] "The git-lfs switch is https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/904239" [integration/config] - 10https://gerrit.wikimedia.org/r/904551 (https://phabricator.wikimedia.org/T333465) (owner: 10Hashar) [15:26:14] 10Release-Engineering-Team, 10Security-Team: deploy_security.py should check if user.name and user.email git configs are set - https://phabricator.wikimedia.org/T333572 (10Lucas_Werkmeister_WMDE) [15:40:29] 10Phabricator, 10Release-Engineering-Team (Radar), 10Patch-For-Review, 10User-brennen: Show "other assignee" avatar on tasks in workboard - https://phabricator.wikimedia.org/T329974 (10brennen) [15:40:47] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Cannot access workboard: Unhandled Exception: Undefined variable: other_assignees - https://phabricator.wikimedia.org/T332234 (10brennen) 05In progress→03Resolved This is redeployed with the fix. [15:41:06] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10User-brennen: Phabricator deployment 2023-03-30 - https://phabricator.wikimedia.org/T333516 (10brennen) [15:41:22] 10Phabricator, 10Release-Engineering-Team (Radar), 10Patch-For-Review, 10User-brennen: Show "other assignee" avatar on tasks in workboard - https://phabricator.wikimedia.org/T329974 (10brennen) 05Open→03Resolved Redeployed with fix. [15:41:39] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Developer Productivity, 10Upstream, 10User-brennen: "Call to phutil_nonempty_string() expected null or a string, got: int." when attempting to view Subversion repos - https://phabricator.wikimedia.org/T310936 (10brennen) 05Open→03Resolved [15:43:31] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team, 10User-brennen: GitLab MR widget sometimes errors out on a missing index - https://phabricator.wikimedia.org/T332607 (10brennen) 05Open→03In progress p:05Triage→03Medium I //think// this should be fixed after this morning's Phabric... [15:56:20] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10User-brennen: Phabricator deployment 2023-03-30 - https://phabricator.wikimedia.org/T333516 (10brennen) 05Open→03Resolved [15:58:22] 10Phabricator, 10Release-Engineering-Team (Radar), 10Patch-For-Review, 10User-brennen: Show "other assignee" avatar on tasks in workboard - https://phabricator.wikimedia.org/T329974 (10Jdlrobson) Amazing. [16:06:28] 10Release-Engineering-Team, 10Security-Team: deploy_security.py should check if user.name and user.email git configs are set - https://phabricator.wikimedia.org/T333572 (10Lucas_Werkmeister_WMDE) Suggested behavior: make this an error during dry-run, to make sure the user notices it; but skip the check on `--r... [16:31:31] (03PS1) 10Hashar: Extract and deploy upstream plugins [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/904575 [16:43:05] (03CR) 10Hashar: "In the child change https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/904575 I am adding some more plugins tracked by git lfs." [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/904239 (https://phabricator.wikimedia.org/T333465) (owner: 10Hashar) [16:46:23] (03CR) 10Hashar: "The parent change migrates Gerrit deployment from git-fat to git-lfs. Jaime and I successfully used it for the Gitlab jenkins-deploy repo." [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/904575 (owner: 10Hashar) [16:46:59] (03CR) 10Hashar: [C: 03+2] "With git-lfs, I have proposed to add the bundled Gerrit plugins in the deployment repository again https://gerrit.wikimedia.org/r/c/operat" [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/699035 (https://phabricator.wikimedia.org/T278990) (owner: 10Ahmon Dancy) [16:54:06] 10GitLab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10xcollazo) [17:00:09] 10GitLab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) usage right now in / is 40% and in /var/lib/docker it's 85% After running an `apt-get clean` usage in / is down to 27%. seems like something already cleaned up meanwhile. still of co... [17:02:00] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) [17:04:06] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) I wonder if this happens to be the runner from: ` < dancy> We enabled another instance wide Gitlab Runner that accepts untagged job... [17:06:08] 10Continuous-Integration-Config, 10Pywikibot, 10Pywikibot-tests: Jenkins output for pywikibot job is hard to read - https://phabricator.wikimedia.org/T117570 (10JJMC89) The [[ https://flake8.pycqa.org/en/6.0.0/user/options.html#cmdoption-flake8-color | documentation ]] says that the color option cannot be sp... [17:36:28] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10dancy) >>! In T333586#8742789, @Dzahn wrote: > I wonder if this happens to be the runner from: > > > ` > < dancy> We enabled another insta... [17:43:16] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) Alright, thanks dancy! So, the remaining space in /var/lib/docker is right now about 5.9GB. The `clear-docker-cache.timer` ran 42 m... [17:57:53] 10Continuous-Integration-Config, 10Moderator-Tools-Team, 10PageTriage, 10Growth-Team (Current Sprint), 10Patch-For-Review: Add PageTriage to gated extensions - https://phabricator.wikimedia.org/T333534 (10Novem_Linguae) For my own learning, is the "gate" documented anywhere? Is that just a fancy term for... [18:27:47] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10xcollazo) Seems like I just happened to run my pipeline on an unlucky window? Does it make sense to consider running the cron jobs more fre... [18:58:44] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10hashar) You can get a view of the partitions free space at https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-b... [19:08:11] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) >>! In T333586#8743072, @xcollazo wrote: > Does it make sense to consider running the cron jobs more frequently... [19:35:39] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10hashar) For the immediate action one can delete the volumes on the runner and that solves this task. Among the larges... [19:47:20] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10xcollazo) >>! In T333586#8743236, @hashar wrote: > (I am pretty sure airflows-dag uses conda which duplicate a large a... [21:08:53] 10Continuous-Integration-Config, 10Pywikibot, 10Patch-For-Review, 10Pywikibot-tests: Jenkins output for pywikibot job is hard to read - https://phabricator.wikimedia.org/T117570 (10hashar) >>! In T117570#8742792, @JJMC89 wrote: > The [[ https://flake8.pycqa.org/en/6.0.0/user/options.html#cmdoption-flake8-c... [21:24:09] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Patch-For-Review, 10User-brennen: Rename mainline branch from "master" to "main" in GitLab:repos/releng/release - https://phabricator.wikimedia.org/T329770 (10thcipriani) > (Internal Release-Engineering-Team) update tr... [21:25:37] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T330208 (10thcipriani) 05Open→03Resolved [21:26:41] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Patch-For-Review, 10User-brennen: Rename mainline branch from "master" to "main" in GitLab:repos/releng/release - https://phabricator.wikimedia.org/T329770 (10jeena) >>! In T329770#8743492, @thcipriani wrote: >> (Inter... [21:52:15] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) ` root@runner-1030:/var/lib/docker/volumes# for volume in $(du -hs * | grep G | cut -d "G" -f2 | xargs); do ls... [22:02:14] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) >>! In T333586#8743236, @hashar wrote: > For the immediate action one can delete the volumes on the runner and... [22:04:16] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) 05Open→03Resolved a:03Dzahn available disk space in /var/lib/docker is back to: used: 21G available: 17... [22:05:39] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T333586 (10Dzahn) well there is still https://gerrit.wikimedia.org/r/c/operations/puppet/+/904616 as a follow-up action.. so mayb... [22:52:16] 10GitLab (Infrastructure), 10SRE, 10ops-codfw, 10serviceops-collab: Install additional SSDs on gitlab2003.wikimedia.org (B5) - https://phabricator.wikimedia.org/T333304 (10Papaul) @Jelto the 2 disks are in place in gitlab2003 [22:52:54] 10GitLab (Infrastructure), 10SRE, 10ops-codfw, 10serviceops-collab: Install additional SSDs on gitlab2003.wikimedia.org (B5) - https://phabricator.wikimedia.org/T333304 (10Papaul) a:03Jelto [23:32:10] PROBLEM - Check systemd state on doc1002 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc1003.eqiad.wmnet.service,rsync-doc-doc2001.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state