[00:35:18] 10Release-Engineering-Team, 10Scap: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10dancy) a:03dancy [02:36:20] is there anyone able to review a patch for Selenium tests? currently they're blocking merges in the CodeMirror extension https://gerrit.wikimedia.org/r/c/mediawiki/core/+/874951 [02:53:19] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T325581 (10matmarex) [09:16:59] (03PS7) 10Hashar: Display Zuul status of jobs for a change in Gerrit [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/859127 (https://phabricator.wikimedia.org/T214068) [09:58:18] (03PS1) 10Lucas Werkmeister (WMDE): zuul: Link to report_url (/console) if available [integration/docroot] - 10https://gerrit.wikimedia.org/r/878860 [09:59:02] (03CR) 10Lucas Werkmeister (WMDE): "Just a suggestion :) thoughts?" [integration/docroot] - 10https://gerrit.wikimedia.org/r/878860 (owner: 10Lucas Werkmeister (WMDE)) [10:21:55] For the record, I just gave the mediawiki group the right to do +2 verified and "submit" on the flow extension, in order to work around dependency issues where unit tests are preventing any patches from being merged into flow [10:58:43] (03CR) 10Hashar: [C: 03+1] "+ Timo since wrote all the Zuul status templating." [integration/docroot] - 10https://gerrit.wikimedia.org/r/878860 (owner: 10Lucas Werkmeister (WMDE)) [17:04:33] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Automate GitLab version upgrade process - https://phabricator.wikimedia.org/T323569 (10Jelto) [17:04:46] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Automate GitLab version upgrade process - https://phabricator.wikimedia.org/T323569 (10Jelto) The `sre.gitlab.upgrade` cookbook was merged and is available on cumin hosts now. Some minor errors were fixed with laser fast reviews from Riccard... [17:28:45] (03PS2) 10Krinkle: zuul: Link to report_url if available [integration/docroot] - 10https://gerrit.wikimedia.org/r/878860 (owner: 10Lucas Werkmeister (WMDE)) [17:28:52] (03CR) 10Krinkle: [C: 03+1] "LGTM." [integration/docroot] - 10https://gerrit.wikimedia.org/r/878860 (owner: 10Lucas Werkmeister (WMDE)) [17:29:17] (03CR) 10Krinkle: [C: 03+1] zuul: Link to report_url if available (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/878860 (owner: 10Lucas Werkmeister (WMDE)) [17:47:22] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T325581 (10matmarex) [17:47:53] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Holiday Leftovers 🥡), 10User-brennen: Build a widget to display GitLab changes on related Phabricator tasks - https://phabricator.wikimedia.org/T324149 (10brennen) [17:48:42] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10MediaWiki Train Development Environment, 10Patch-For-Review: Migrate train-dev to GitLab - https://phabricator.wikimedia.org/T323225 (10thcipriani) 05Open→03Resolved [17:49:05] 10GitLab (Project Migration), 10Release-Engineering-Team (Holiday Leftovers 🥡), 10User-brennen: Migrate mediawiki/tools/release/ to GitLab - https://phabricator.wikimedia.org/T290260 (10thcipriani) [17:50:26] 10GitLab (Integrations), 10GitLab-Test, 10Phabricator, 10Release-Engineering-Team (Holiday Leftovers 🥡), and 2 others: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10thcipriani) [17:50:33] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Holiday Leftovers 🥡), 10User-brennen: Comment on Phabricator tasks for new, merged, and abandoned changes on GitLab - https://phabricator.wikimedia.org/T324150 (10thcipriani) [17:51:17] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try Reggie for buildkit caching - https://phabricator.wikimedia.org/T323149 (10thcipriani) 05Open→03Resolved a:03dduvall [17:51:21] 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10Scap: Ensure efficient Gitlab CI operations for scap - https://phabricator.wikimedia.org/T323140 (10thcipriani) [17:51:29] 10Release-Engineering-Team (Holiday Leftovers 🥡): CasC configuration for Releasing Jenkins instance - https://phabricator.wikimedia.org/T323909 (10thcipriani) [17:51:39] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Holiday Leftovers 🥡): Automate integration Jenkins deployment and config changes - https://phabricator.wikimedia.org/T319406 (10thcipriani) [17:52:45] 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10Scap: Ensure efficient Gitlab CI operations for scap - https://phabricator.wikimedia.org/T323140 (10dancy) 05Open→03Resolved a:03dancy Scap Gitlab CI is working well, close enough to the desired time targets. [17:52:47] 10Release-Engineering-Team (Holiday Leftovers 🥡): Add Gitlab JWT support to Reggie - https://phabricator.wikimedia.org/T323394 (10thcipriani) [17:53:36] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Holiday Leftovers 🥡), 10User-brennen: Add DigitalOcean resource monitoring for cloud runner nodes - https://phabricator.wikimedia.org/T308615 (10thcipriani) [17:53:49] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Holiday Leftovers 🥡), 10serviceops-collab, 10User-brennen: Provision untrusted instance-wide GitLab job runners to handle user-level projects and merge requests from forks - https://phabricator.wikimedia.org/T297426 (10thcipriani) [18:24:37] 10Gerrit, 10Release-Engineering-Team (Holiday Leftovers 🥡), 10SRE-OnFire, 10serviceops-collab, and 2 others: gerrit1001 running out of space on / - https://phabricator.wikimedia.org/T323262 (10thcipriani) [18:25:51] (03CR) 10Arlolra: [C: 03+2] Bump puppeteer version + npm audit fix [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/870695 (owner: 10Subramanya Sastry) [18:27:06] 10Release-Engineering-Team (Holiday Leftovers 🥡), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T325581 (10thcipriani) [18:27:13] (03Merged) 10jenkins-bot: Bump puppeteer version + npm audit fix [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/870695 (owner: 10Subramanya Sastry) [18:36:54] (03CR) 10Arlolra: [C: 03+2] cache_purge_adaptor: Follow redirects while purging [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/869343 (owner: 10Subramanya Sastry) [18:37:38] (03Merged) 10jenkins-bot: cache_purge_adaptor: Follow redirects while purging [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/869343 (owner: 10Subramanya Sastry) [19:05:01] Project beta-update-databases-eqiad build #64352: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/64352/ [19:33:56] I wrote httpbb tests for Phabricator and the other domains hosted on a Phab servers, which include bugzilla.wikimedia.org, git.wikimedia.org .... and various rules for diffusion [19:34:00] https://gerrit.wikimedia.org/r/c/operations/puppet/+/878203/3/modules/profile/files/httpbb/phabricator/test_phabricator.yaml [19:34:18] because I feel much better with this before merging new rewrite rules [19:34:41] see those special cases for the complicated rules once written by DannyB ... [19:56:57] thx for heads up on those tests. [19:58:44] [deploy1002:~] $ httpbb --hosts phab1004.eqiad.wmnet,phab2002.codfw.wmnet /srv/deployment/httpbb-tests/phabricator/test_phabricator.yaml [19:58:47] Sending to 2 hosts... [19:58:53] from deployment servers [19:58:53] (03CR) 10Arlolra: [C: 03+2] Followup to 6fb18f41: DT adds multiple comments now [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/868455 (owner: 10Subramanya Sastry) [19:59:31] and oh look.. right away we find a difference [19:59:31] (03Merged) 10jenkins-bot: Followup to 6fb18f41: DT adds multiple comments now [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/868455 (owner: 10Subramanya Sastry) [19:59:41] on phab2002 one thing is 500 instead of 302 [21:02:52] (03CR) 10Subramanya Sastry: "Not needed anymore. Will abandon." [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/869339 (owner: 10Subramanya Sastry) [21:03:21] (03Abandoned) 10Subramanya Sastry: Temporary Cite CSS fixes [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/869339 (owner: 10Subramanya Sastry) [21:04:04] (03CR) 10Subramanya Sastry: [C: 04-1] "I have updated the cite css generation script since I uploaded the patch - will resubmit patch with updated css rules." [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/869335 (owner: 10Subramanya Sastry) [21:14:07] (03PS2) 10Subramanya Sastry: Migrate Parsoid HTML fetch from RESTBase to the REST API [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/876258 [21:14:09] (03PS8) 10Subramanya Sastry: Add script-generated Cite CSS [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/869335 [23:20:48] (03CR) 10Arlolra: [C: 03+2] Migrate Parsoid HTML fetch from RESTBase to the REST API (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/876258 (owner: 10Subramanya Sastry) [23:21:26] (03Merged) 10jenkins-bot: Migrate Parsoid HTML fetch from RESTBase to the REST API [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/876258 (owner: 10Subramanya Sastry) [23:21:46] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) I noticed we were nowadays only at "9/10" instances quota-wise.. so I jumped on that and created gitlab-prod-1002 just now. ..which filled up the quota but we got away... [23:27:05] (03CR) 10Subramanya Sastry: Migrate Parsoid HTML fetch from RESTBase to the REST API (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/876258 (owner: 10Subramanya Sastry) [23:27:39] 10Release-Engineering-Team (Holiday Leftovers 🥡), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T325581 (10Jdlrobson) [23:29:10] (03CR) 10Subramanya Sastry: Migrate Parsoid HTML fetch from RESTBase to the REST API (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/876258 (owner: 10Subramanya Sastry) [23:29:25] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) instance created, puppetmaster changed via Hiera: ` -server = puppetmaster.cloudinfra.wmflabs.org -ca_server = puppetmaster.cloudinfra.wmflabs.org +server = puppetmas... [23:30:57] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) and FAIL.. ` Warning: Unable to fetch my node definition, but the agent run will continue: Warning: SSL_connect returned=1 errno=0 state=error: certificate verify fai... [23:33:43] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) this is fixed by `rm -rf /var/lib/puppet/ssl` and running puppet again to create new requests. keeps coming up every time we switch masters, more or less, since years:... [23:55:58] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) next issue: ` Setting up gitlab-ce (15.5.7-ce.0) ... It looks like there was a problem with public attributes; run gitlab-ctl reconfigure manually to fix. dpkg: error... [23:57:00] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) running `gitlab-ctl reconfigure` manually results in: ` Running handlers: [2023-01-11T23:56:28+00:00] ERROR: Running exception handlers There was an error running git... [23:58:15] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) This does seem like a general issue with Letsencrypt / acme_chief in cloud as in the past.