[02:01:09] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Seddon) I was previously grant the rights in T156480 but I migrated to a different account. Could this account be added to the new group? Many thanks in advance! [10:04:38] 10Phabricator: Phabricator file deletion request - https://phabricator.wikimedia.org/T336928 (10Niridya) [10:17:42] 10Phabricator, 10Wikimedia-Hackathon-2023: Hacking Wikimedia Phabricator / Phorge - https://phabricator.wikimedia.org/T336167 (10valerio.bozzolan) [10:22:55] 10Phabricator, 10Wikimedia-Hackathon-2023: Hacking Wikimedia Phabricator / Phorge - https://phabricator.wikimedia.org/T336167 (10Tgr) Do you think it is worth the time (as in, is there a chance of feature suggestions actually getting picked up)? TBH my impression was that the project has been on life support s... [10:23:58] 10Phabricator, 10Wikimedia-Hackathon-2023: Hacking Wikimedia Phabricator / Phorge - https://phabricator.wikimedia.org/T336167 (10RhinosF1) Phorge is definitely not on life support which is the community fork of Phab that formed since then. [12:09:05] 10MediaWiki-Releasing, 10Security: Backport security fixes to stable + LTS [non-bundled] extension branches - https://phabricator.wikimedia.org/T108734 (10Tgr) [12:32:02] 10Phabricator, 10Wikimedia-Hackathon-2023: Hacking Wikimedia Phabricator / Phorge - https://phabricator.wikimedia.org/T336167 (10Tgr) The 2017 developer wishlist had a [[https://www.mediawiki.org/wiki/Developer_Wishlist/2017/Tools_(Phabricator,_Gerrit,_etc.)|tooling section]] which was largely about Phabricato... [13:37:14] 10Phabricator, 10Wikimedia-Hackathon-2023: Hacking Wikimedia Phabricator / Phorge - https://phabricator.wikimedia.org/T336167 (10valerio.bozzolan) >>! In T336167#8861489, @Tgr wrote: > Do you think it is worth the time (as in, is there a chance of feature suggestions actually getting picked up)? Believe me or... [14:30:25] 10Project-Admins: Create a #wmcs-superset tag - https://phabricator.wikimedia.org/T336940 (10rook) [14:31:58] 10Project-Admins: Create a #openstack-magnum - https://phabricator.wikimedia.org/T336941 (10rook) [15:58:57] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T330215 (10brennen) 05Open→03In progress a:05dancy→03brennen Taking over train ops for the day. [17:23:39] Project beta-code-update-eqiad build #444302: 15ABORTED in 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/444302/ [18:31:05] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T330215 (10brennen) 05In progress→03Open Changing status back here since it messes with `scap train`. [18:41:08] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T330215 (10brennen) [18:41:39] 10GitLab (Infrastructure), 10SRE, 10ops-eqiad, 10serviceops-collab: gitlab-runner1003 is not coming back online - https://phabricator.wikimedia.org/T336737 (10Jclark-ctr) @Jelto Updated Firmware on Server additionally and performed reboot still boots properly [18:48:03] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T330215 (10brennen) [20:47:57] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T330215 (10Nux) Not sure if related but I see WD is on 1.41.0-wmf.9 (bacc43a) https://www.wikidata.org/wiki/Special:Version... [20:52:10] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T330215 (10brennen) > So possibly a blocker? I see the original stack trace there is for wmf.8, so I think that one is unli... [20:55:05] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T330215 (10brennen) > I see the original stack trace there is for wmf.8, so I think that one is unlikely to be a result of t... [21:26:04] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T330215 (10brennen) I'm going ahead with the train to group2, since that won't change the state of Wikidata, but if T336952... [21:35:42] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T330215 (10Umherirrender) [21:45:50] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T330215 (10brennen) [22:22:41] zuul-merger is down for a short time, maintenance on contint2001 [22:23:26] PROBLEM - zuul_merger_service_running on contint2001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args bin/zuul-merger https://www.mediawiki.org/wiki/Continuous_integration/Zuul [22:24:09] ACKNOWLEDGEMENT - zuul_merger_service_running on contint2001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args bin/zuul-merger daniel_zahn maintenance work https://www.mediawiki.org/wiki/Continuous_integration/Zuul [22:25:13] is running sudo chown -R 923:923 /srv/zuul/git /var/lib/zuul but it just takes longer than anticipated [22:25:18] large tree [22:52:48] RECOVERY - zuul_merger_service_running on contint2001 is OK: PROCS OK: 1 process with regex args bin/zuul-merger https://www.mediawiki.org/wiki/Continuous_integration/Zuul [22:59:12] PROBLEM - zuul_gearman_service on contint2001 is CRITICAL: connect to address 127.0.0.1 and port 4730: Connection refused https://www.mediawiki.org/wiki/Continuous_integration/Zuul [22:59:22] PROBLEM - zuul_service_running on contint2001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args bin/zuul-server https://www.mediawiki.org/wiki/Continuous_integration/Zuul [23:00:14] ACKNOWLEDGEMENT - zuul_gearman_service on contint2001 is CRITICAL: connect to address 127.0.0.1 and port 4730: Connection refused daniel_zahn maintenance https://www.mediawiki.org/wiki/Continuous_integration/Zuul [23:00:14] ACKNOWLEDGEMENT - zuul_service_running on contint2001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args bin/zuul-server daniel_zahn maintenance https://www.mediawiki.org/wiki/Continuous_integration/Zuul [23:02:20] RECOVERY - zuul_gearman_service on contint2001 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 4730 https://www.mediawiki.org/wiki/Continuous_integration/Zuul [23:02:28] RECOVERY - zuul_service_running on contint2001 is OK: PROCS OK: 2 processes with regex args bin/zuul-server https://www.mediawiki.org/wiki/Continuous_integration/Zuul [23:02:42] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T330215 (10brennen) [23:05:02] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T330215 (10brennen) Nearing end of workday here and I have obligations away from the computer, but I'm treating T336952 as a... [23:42:21] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, 10Patch-For-Review: contint2002 service implementation tracking - https://phabricator.wikimedia.org/T324659 (10Dzahn) after deploying the change above carefully on all 3 contint* servers, stopping services, running manual chown commands ,... [23:46:43] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, 10Patch-For-Review: contint2002 service implementation tracking - https://phabricator.wikimedia.org/T324659 (10Dzahn) - disable puppet - stop services - chown -R 923:923 /srv/zuul/git /var/lib/zuul - chown -R 923:923 /var/log/zuul_repack/...