[04:45:08] 10Continuous-Integration-Infrastructure, 10Jenkins: HTTP ERROR 403 No valid crumb was included in the request - https://phabricator.wikimedia.org/T327988 (10TheresNoTime) [04:49:10] 10Release-Engineering-Team (Radar), 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Ladsgroup) This is somewhat important for mw-on-k8s so even if the perf gai... [05:08:57] 10GitLab (CI & Job Runners): GitLab CI: Could not resolve host: gitlab.wikimedia.org - https://phabricator.wikimedia.org/T327989 (10Legoktm) [05:12:42] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10taavi) >>! In T318521#8559071, @Dzahn wrote: >>>! In T318521#8551236, @taavi wrote: >>> @taavi are you able to re-create that setup for `gitlab-prod-1002` in `devtools` projec... [08:01:57] the changelog for this week's train is missing: https://www.mediawiki.org/wiki/MediaWiki_1.40/wmf.20/Changelog [08:45:34] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Cloud-VPS (Debian Stretch Deprecation): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Peachey88) [08:51:34] PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [08:53:02] RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 65172 bytes in 8.989 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [09:00:28] PROBLEM - Gerrit Health Check on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [09:00:29] PROBLEM - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [09:00:48] PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [09:01:37] RECOVERY - Gerrit Health Check on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 973 bytes in 4.196 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [09:01:37] RECOVERY - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is OK: OK - Certificate gerrit.wikimedia.org will expire on Wed 01 Mar 2023 09:47:05 PM GMT +0000. https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [09:02:03] RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 66291 bytes in 7.481 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [09:20:27] PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [09:21:41] RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 69123 bytes in 0.055 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [09:25:31] Gerrit complained about Internal error during upload-pack from Repository[/srv/gerrit/git/operations/puppet.git] , same as yesterday, then recovered. I have the full error mesage, if someone knows how to read it [09:32:40] if you file a task, has.har will probably look at it [09:39:24] 10Gerrit: Gerrit randomly pauses - https://phabricator.wikimedia.org/T328003 (10SLyngshede-WMF) [09:41:24] 10Gerrit, 10Release-Engineering-Team: Gerrit randomly pauses - https://phabricator.wikimedia.org/T328003 (10SLyngshede-WMF) [09:59:04] Krinkle: turns out Gerrit went wild this morning for some reason :) [09:59:16] so what ever you observed earlier this week was probably an early sign [10:00:05] 10Gerrit, 10Release-Engineering-Team: Gerrit randomly pauses - https://phabricator.wikimedia.org/T328003 (10hashar) [10:01:58] 10Gerrit, 10Release-Engineering-Team: Gerrit randomly pauses - https://phabricator.wikimedia.org/T328003 (10hashar) [10:02:09] slyngs: thank you for the task filing about Gerrit, I will dig in it [10:02:45] hashar: Thank you, it recovers automatically, but not really desired behavior :-) [10:46:00] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: Align the GitLab runner tags - https://phabricator.wikimedia.org/T325069 (10Jelto) To sum up the current state: Trusted runners use tag `trusted` now Shared runners in WMCS use tag `wmcs` now (but accept untagged jobs) I'd lik... [10:46:15] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: Align the GitLab runner tags - https://phabricator.wikimedia.org/T325069 (10Jelto) [10:49:45] PROBLEM - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [10:49:55] PROBLEM - Gerrit Health Check on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [10:50:17] PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [10:51:03] RECOVERY - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is OK: OK - Certificate gerrit.wikimedia.org will expire on Wed 01 Mar 2023 09:47:05 PM GMT +0000. https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [10:51:11] RECOVERY - Gerrit Health Check on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 972 bytes in 0.027 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [10:51:37] RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 61922 bytes in 0.042 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [11:01:59] PROBLEM - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [11:02:07] PROBLEM - Gerrit Health Check on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [11:02:34] PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [11:03:27] RECOVERY - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is OK: OK - Certificate gerrit.wikimedia.org will expire on Wed 01 Mar 2023 09:47:05 PM GMT +0000. https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [11:03:37] RECOVERY - Gerrit Health Check on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 973 bytes in 0.037 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [11:04:01] RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 66644 bytes in 0.046 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [11:17:00] the source of those Gerrit errors has been handled. So should be fixed [11:39:25] <3 [13:10:42] ACKNOWLEDGEMENT - Host gerrit2002 is DOWN: PING CRITICAL - Packet loss = 100% amusso reboot! [14:26:10] 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team, 10performance-team-onboarding: Make it possible to run Fresnel on MediaWiki extension repos - https://phabricator.wikimedia.org/T220162 (10larissagaulia) [14:49:42] 10Continuous-Integration-Infrastructure, 10Jenkins, 10SRE, 10SRE-Access-Requests, and 2 others: New Keyholder identity for RelEng Jenkins service - https://phabricator.wikimedia.org/T324014 (10jnuche) [15:30:26] jnuche: I am going to upgrade the Jenkins :) [15:30:44] I kind of forgot about it, I don't thik I can do it tonight and well tomorrow is friday [15:30:52] hashar: spoffice? [15:31:02] what? [15:31:16] let's join the spontanoffice call? [15:31:34] (it's the same as the watercooler) [15:31:42] hhhhooo [15:31:50] yeah let me relocate to another room [15:31:58] (https://meet.google.com/sut-zxhw-jqy) [15:44:55] 10Release-Engineering-Team (Radar), 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10dancy) >>! In T99740#8559738, @Ladsgroup wrote: > This is somewhat importan... [15:57:03] 10Continuous-Integration-Infrastructure, 10Jenkins: Upgrade Jenkins to latest LTS 2.375.2 - https://phabricator.wikimedia.org/T326531 (10hashar) 05Open→03Resolved a:03hashar @Jaime and I paired the upgrade. https://debmonitor.wikimedia.org/packages/jenkins says all 4 hosts have `2.375.2` [16:05:43] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10CirrusSearch, 10Discovery-Search: Consider adding "official" docker.elastic.co images to the list of allowed images for gitlab runners - https://phabricator.wikimedia.org/T327519 (10bd808) 7.10.2 is the final Elasticsearch version released under an... [16:33:28] 10GitLab, 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱): Make a tool to convert .pipeline/config.yaml to .gitlab-ci.yaml - https://phabricator.wikimedia.org/T327332 (10dancy) a:03dancy [16:34:45] 10GitLab, 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱), 10serviceops-collab: Convert runner-1030.gitlab-runners.eqiad1.wikimedia.cloud to an instance-wide shared runner - https://phabricator.wikimedia.org/T327949 (10Jelto) [17:02:16] @6 [17:02:23] sorry [17:43:45] 10GitLab: Support passing environment variables to kokkuri's build-and-run-image - https://phabricator.wikimedia.org/T328054 (10dancy) [17:44:09] 10GitLab, 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱): Support passing environment variables to kokkuri's build-and-run-image - https://phabricator.wikimedia.org/T328054 (10dancy) [19:32:02] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: gdi - https://phabricator.wikimedia.org/T327939 (10ntsako) That makes sense. Please add myself, @KCVelaga_WMF, @CMacholan and @YLiou_WMF as owners. [19:37:29] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: gdi - https://phabricator.wikimedia.org/T327939 (10brennen) 05Open→03In progress a:03brennen [19:43:19] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: gdi - https://phabricator.wikimedia.org/T327939 (10brennen) 05In progress→03Resolved @ntsako - I've added you as an owner. I wasn't able to find existing GitLab accounts for the rest of the team. You should be able... [19:45:14] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: gdi - https://phabricator.wikimedia.org/T327939 (10ntsako) Thanks a lot for your help @brennen, I'll advise them to create Gitlab accounts so that they can be added. [19:53:57] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: machine-learning - https://phabricator.wikimedia.org/T325051 (10brennen) 05Open→03Resolved a:03brennen Created: https://gitlab.wikimedia.org/repos/machine-learning > Owners should probably be Chris, Luca and me, a... [20:36:06] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: Align the GitLab runner tags - https://phabricator.wikimedia.org/T325069 (10bd808) >>! In T325069#8560470, @Jelto wrote: > Shared runners in public cloud use `cloud` (and `kubernetes`) currently. I'm not sure if thee is a strong... [22:45:29] (03PS1) 10Zabe: zuul: Add Superpes to allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/884127 [22:53:25] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.40.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T325584 (10Jdlrobson)