[00:37:22] 10Phabricator (Upstream), 10Accessibility, 10Upstream: Improve Phabricator's assistive technology support - https://phabricator.wikimedia.org/T109 (10Graham87) [00:44:08] stopping jenkins on contint2001 (main CI server) [00:44:16] short downtime [00:49:53] 10Phabricator (Upstream), 10Accessibility, 10Upstream: Improve Phabricator's assistive technology support - https://phabricator.wikimedia.org/T109 (10Graham87) I'm not a heavy Phabricator user but I can't think of anything that needs to be fixed accessibility-wise these days. I use the latest version of JAWS... [01:07:06] not so short .. but ongoing.. [01:20:04] contint2001 - jenkins started again after maintenance and changing UID/GID [01:20:35] some little pitfalls, like symlinks from /var/lib/jenkins to /srv/jenkins and without -h chmod won't change symlinks [01:20:46] and just that /srv/jenkins is so large on contint2001 but only there [01:21:26] also one extra command needed to avoid ending up with some files owned jenkins:499. will add on gerrit change [01:21:39] should be all fixed and done now. maintenance over [01:35:14] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, 10Patch-For-Review: contint2002 service implementation tracking - https://phabricator.wikimedia.org/T324659 (10Dzahn) after carefully deploying the patch above to change jenkins UID/GID, following the instructions, changing file ownership... [01:38:23] 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab, 10Patch-For-Review, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10Dzahn) Great news and work !:) thank you! [05:31:56] mutante: argh symlinks.... xongrats ont the UID change! [06:55:53] 10Phabricator (Upstream), 10Upstream: Phabricator should suggest possible duplicates when creating a new task - https://phabricator.wikimedia.org/T45 (10valerio.bozzolan) [07:16:23] 10Phabricator (Upstream), 10Upstream: Phabricator should suggest possible duplicates when creating a new task - https://phabricator.wikimedia.org/T45 (10valerio.bozzolan) Another root problem: it's difficult, if not impossible for newcomers, to get a "big picture" after landing on a bug reporting form. That of... [07:29:39] 10Project-Admins: Create project tag for Documentation metrics dashboard - https://phabricator.wikimedia.org/T337363 (10KBach) [07:31:22] 10Phabricator (Upstream), 10Accessibility, 10Upstream: Improve Phabricator's assistive technology support - https://phabricator.wikimedia.org/T109 (10valerio.bozzolan) Wow. @Graham87 This is definitely the most unexpected, relaxing, beautiful comment I could read in the last 6 months. Thanks for your live te... [07:35:54] 10Phabricator (Upstream), 10Accessibility, 10Upstream: Improve Phabricator's assistive technology support - https://phabricator.wikimedia.org/T109 (10Graham87) Yeah the transcript should be in the alt text in a case like this ... that's how I've encountered that sort of thing on Mastodon. Nah it didn't make... [07:47:11] 10Phabricator (Upstream), 10Accessibility, 10Upstream: Improve Phabricator's assistive technology support - https://phabricator.wikimedia.org/T109 (10valerio.bozzolan) >>! In T109#8875540, @Graham87 wrote: > Yeah the transcript should be in the alt text in a case like this ... that's how I've encountered tha... [08:24:02] (03CR) 10Hashar: "I have added some very rough QUnit tests ;)" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/922605 (https://phabricator.wikimedia.org/T332474) (owner: 10Hashar) [08:48:34] 10Continuous-Integration-Infrastructure, 10serviceops-collab, 10Patch-For-Review: Migrate doc hosts to Bullseye - https://phabricator.wikimedia.org/T319477 (10hashar) Sounds good! I will make my best to be around though I got kids incoming at that time. We shall see. A word of caution, there are systemd tim... [08:48:37] (03CR) 10Dreamy Jazz: wm-patch-demo: initial implementation (031 comment) [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/922605 (https://phabricator.wikimedia.org/T332474) (owner: 10Hashar) [08:50:00] (03CR) 10Hashar: wm-patch-demo: initial implementation (031 comment) [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/922605 (https://phabricator.wikimedia.org/T332474) (owner: 10Hashar) [08:50:48] (03PS4) 10Hashar: wm-patch-demo: initial implementation [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/922605 (https://phabricator.wikimedia.org/T332474) [08:51:57] eoghan: good morning, I will do my best to be around for the doc hosts switch :] It is at 10am UTC / 1 hour from now isn't it? [08:52:27] I have added a small remark about the systemd timers that need to be purged by puppet on doc1002 [08:52:42] 10Scap, 10Machine-Learning-Team, 10ORES: scap support for multiple services - https://phabricator.wikimedia.org/T185255 (10elukey) 05Open→03Declined The ML team is focusing on https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing, the replacement of ORES. Please re-open if you feel that something... [08:52:53] there is a potential race condition; Then if some material vanishes as part of the switch, the next build in Jenkins will eventually catch up with reality [08:52:58] it is probably fine [08:53:17] 10Continuous-Integration-Infrastructure, 10serviceops-collab, 10Patch-For-Review: Migrate doc hosts to Bullseye - https://phabricator.wikimedia.org/T319477 (10eoghan) @hashar Thanks for that! Yeah, I'm aware of the timers. I had intended for puppet-agent to be run on both doc1003 and doc1002 at the same time... [08:53:40] hashar: Yep, just saw that. Thanks for catching! I was planning to do both at the same time to make sure the timers were gone. [08:53:53] jnuche: ^ for doc* updates :] [08:54:01] eoghan: yeah sounds good [08:54:20] the whole of CI points to doc.discovery.wmnet or something so it should be fine [08:54:25] hashar: An hour from now is correct, yeah. I can probably start a bit before that, since I don't think this is necessarily time-sensitive or will be user impacting, but I need to drop the kids off first. Will let you know when I'm back and we can see how we're all fixed. [08:54:38] though maybe the WMCS instance would have a cached copy of the DNS resolution for up to whatever the TTL is set [08:54:43] Yep, exactly, I asked jnuche to update the gitlab jobs to point to the discovery endpoint yesterday [08:54:57] \o/ [08:54:58] It's possible, yeah. I'm not hugely concerned I think. [08:55:13] I am not either [08:55:28] we switched it fairly recently and it went smoothly [08:55:37] thanks to the discovery CNAME [08:56:44] eoghan: the new doc publisher is rsyncing to the discovery endpoint and ready for the switch [08:56:58] Wonderful, thanks! [09:21:54] FYI: I'm going to create a new DB table on beta for https://phabricator.wikimedia.org/T336362 [09:24:37] !log Creating new DB table on beta wikishared for the CampaignEvents extension # T336362 [09:24:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:24:40] T336362: Update beta to use the new tracking tools schema - https://phabricator.wikimedia.org/T336362 [09:57:37] 10Project-Admins: Create project tag for Documentation metrics dashboard - https://phabricator.wikimedia.org/T337363 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project has been created: #Documentation-Metrics-Dashboard (In case you need to edit the project or project workboard itself at... [09:58:30] hashar,jnuche: I'm back and just about to get started. Giving one more look over the changes generated by puppet, then I'll kick off. [09:59:14] o/ [09:59:17] eoghan: ack [10:08:52] 10Phabricator: Email sometimes not being sent when a task is created and "other task activity" is not set in user preferences - https://phabricator.wikimedia.org/T182549 (10Aklapper) 05Open→03Declined No replies; assuming this is not an issue anymore. Please reopen if it still is. [10:09:35] !log Switching doc.wikimedia.org from doc1002 -> doc1003, T319477 [10:09:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:09:38] T319477: Migrate doc hosts to Bullseye - https://phabricator.wikimedia.org/T319477 [10:10:42] rsync stopped on doc1002, FYI. Jobs will fail until the migration is complete. [10:16:16] Project mediawiki-core-doxygen-docker build #43535: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/43535/ [10:24:22] 10Phabricator-Bot-Requests, 10Infrastructure-Foundations: update permissions for offboarding user - https://phabricator.wikimedia.org/T333146 (10Aklapper) @jbond: This likely requires a member of the project #acl_sre-team to add the bot as a member to #acl_sre-team via https://phabricator.wikimedia.org/project... [10:25:23] that job failure is expected (rsync is off) [10:26:01] once migrated we can retrigger that job (it is triggered by Jenkins when it polls the mediawiki/core repo) [10:26:31] OK, we're done! [10:26:54] I've switched it over, verified that I can load the doc page, the rsync between host job is running on doc1003, and I've verified that the timers were removed from doc1002. [10:27:46] (03PS1) 10JMeybohm: helm-linter: Bump version to update kubeconform-json-schema [integration/config] - 10https://gerrit.wikimedia.org/r/922812 [10:27:48] (03PS1) 10JMeybohm: Update to helm-linter:0.4.8 [integration/config] - 10https://gerrit.wikimedia.org/r/922813 [10:28:52] hashar|taavi: If you have a minute ^ <3 thanks! [10:29:00] yeah [10:29:20] (03CR) 10Hashar: [C: 03+2] helm-linter: Bump version to update kubeconform-json-schema [integration/config] - 10https://gerrit.wikimedia.org/r/922812 (owner: 10JMeybohm) [10:29:32] jayme: we should onboard you on deploying those changes by yourself :] [10:30:29] (03Merged) 10jenkins-bot: helm-linter: Bump version to update kubeconform-json-schema [integration/config] - 10https://gerrit.wikimedia.org/r/922812 (owner: 10JMeybohm) [10:30:34] eoghan: I have triggered a build of the job that failed earlier : https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/43536/console [10:30:38] ty! [10:30:41] thanks :D [10:30:44] * jayme https://media.giphy.com/media/jUwpNzg9IcyrK/giphy.gif [10:31:40] PROBLEM - Check systemd state on doc1003 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc2001.codfw.wmnet.service,rsync-doc-doc2002.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [10:33:04] ah maybe cause puppet hasn't run on doc2001/doc2002 yet (and doc1003 is not added to iptables there) [10:33:06] eoghan: ^ [10:33:15] Yep, that's right. Fixing it now [10:35:52] jayme: the image is being uploaded after which I will update the jenkins job [10:36:01] (03CR) 10Hashar: [C: 03+2] "== Step 1: building images ==" [integration/config] - 10https://gerrit.wikimedia.org/r/922812 (owner: 10JMeybohm) [10:36:39] (03CR) 10Hashar: [C: 03+2] "Job updated!" [integration/config] - 10https://gerrit.wikimedia.org/r/922813 (owner: 10JMeybohm) [10:36:46] jayme: {success} [10:37:49] (03Merged) 10jenkins-bot: Update to helm-linter:0.4.8 [integration/config] - 10https://gerrit.wikimedia.org/r/922813 (owner: 10JMeybohm) [10:38:46] and base on https://integration.wikimedia.org/ci/job/publish-to-doc/ , that is the sole job that failed \o/ [10:39:18] RECOVERY - Check systemd state on doc1003 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [10:41:46] eoghan: I can still publish successfully from gitlab and newly generated docs show up on `doc1003.eqiad.wmnet` [10:41:54] everything looks healthy on the gitlab side [10:42:05] thanks for taking care of the migration :) [10:43:22] Yippee, build fixed! [10:43:22] Project mediawiki-core-doxygen-docker build #43536: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/43536/ [10:44:55] Awesome, happy days [10:45:13] 10Continuous-Integration-Infrastructure, 10serviceops-collab, 10Patch-For-Review: Migrate doc hosts to Bullseye - https://phabricator.wikimedia.org/T319477 (10hashar) The CI job publishing is https://integration.wikimedia.org/ci/job/publish-to-doc/ and the sole failure it had was for the job which generates... [10:45:30] eoghan: I have wrote a summary for the jenkins job and validate the doc switch works for it [10:45:39] with gitlab verifying the gitlab side, I think we are set :] [10:45:48] c o n g r a t u l a t i o n s! [10:46:16] Excellent. doc is now running on bullseye. I'm going to shut down the two old buster hosts this afternoon and decommission them later this week. Will update the ticket. [10:46:26] Thanks both for your help on this [10:46:34] then if I get it right we will have to decom doc1002 and doc2001 [10:46:40] \o/ [10:46:57] and I don't think there is anything to salvage from them. They are entirely in puppet or in scap deployed git repo [10:47:07] short of the docs in /srv/doc which got rsynced [10:47:19] so they can be wiped :] [10:47:22] thanks ! [10:47:50] I am off for the afternoon, going to pour concrete. I will be back later though :] [10:50:12] Enjoy! [10:51:18] 10Continuous-Integration-Infrastructure, 10serviceops-collab, 10Patch-For-Review: Migrate doc hosts to Bullseye - https://phabricator.wikimedia.org/T319477 (10eoghan) The migration went smoothly. I'm going to remove the two buster hosts from service this afternoon, and we can decommission them later. [11:30:22] 10Phabricator: Phabricator monthly email: Only list Differential usage statistics for created and updated patchsets - https://phabricator.wikimedia.org/T337382 (10Aklapper) p:05Triage→03Low [11:36:57] thanks hashar! [11:47:20] hashar: when i add api mocha tests to a repo, do I need to do anything to make them run in CI? Or does it just happen? [12:25:12] (03PS1) 10Daniel Kinzler: Enable API tests for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/922833 [12:25:19] (03CR) 10CI reject: [V: 04-1] Enable API tests for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/922833 (owner: 10Daniel Kinzler) [12:27:26] (03CR) 10Daniel Kinzler: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/922833 (owner: 10Daniel Kinzler) [12:27:41] hashar: I figured it out :) [12:35:59] 10Phabricator, 10Developer-Advocacy (Apr-Jun 2023): Automate SQL queries for quarterly Phabricator statistics/metrics for Technical Community Newsletter - https://phabricator.wikimedia.org/T337387 (10Aklapper) p:05Triage→03Medium [12:36:03] 10Phabricator, 10Developer-Advocacy: Automate SQL queries for annual end-of-year Phabricator statistics/metrics on wikitech-l@ - https://phabricator.wikimedia.org/T337388 (10Aklapper) p:05Triage→03Low [12:58:52] 10Continuous-Integration-Config, 10Toolforge Build Service, 10cloud-services-team, 10Cloud-Services-Origin-Team, and 2 others: [tbs] Set up CI for cloud/toolforge/buildpacks repository - https://phabricator.wikimedia.org/T265685 (10dcaro) 05Open→03Resolved a:03dcaro We don't use this repository anymore [13:23:57] 10Release-Engineering-Team (They Live 🕶️🧟), 10Scap: Handle spurious failures in Scap's self-installer - https://phabricator.wikimedia.org/T337394 (10jnuche) [13:24:42] 10Release-Engineering-Team (They Live 🕶️🧟), 10Scap: Handle spurious failures in Scap's self-installer - https://phabricator.wikimedia.org/T337394 (10jnuche) 05Open→03In progress a:03jnuche [13:25:00] (03CR) 10Esanders: [C: 03+1] "Thanks!" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/922605 (https://phabricator.wikimedia.org/T332474) (owner: 10Hashar) [13:25:56] 10Release-Engineering-Team (They Live 🕶️🧟), 10Scap: Handle spurious failures in Scap's self-installer - https://phabricator.wikimedia.org/T337394 (10jnuche) [13:31:01] (03Abandoned) 10Aklapper: Review access change [sandbox] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/904930 (owner: 10Sabariya) [13:56:40] (03PS5) 10Hashar: wm-patch-demo: initial implementation [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/922605 (https://phabricator.wikimedia.org/T332474) [13:57:06] (03CR) 10Hashar: "I forgot to git add `test/wm-patch-demo.js` :/" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/922605 (https://phabricator.wikimedia.org/T332474) (owner: 10Hashar) [13:57:51] 10Phabricator, 10Security-Team, 10SecTeam-Processed, 10Security: Audit members of acl*security for more than 12 months of no activity (May 2024) - https://phabricator.wikimedia.org/T337305 (10sbassett) [14:12:14] (03CR) 10Hashar: [C: 03+2] wm-patch-demo: initial implementation [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/922605 (https://phabricator.wikimedia.org/T332474) (owner: 10Hashar) [14:12:54] (03Merged) 10jenkins-bot: wm-patch-demo: initial implementation [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/922605 (https://phabricator.wikimedia.org/T332474) (owner: 10Hashar) [14:17:51] 10Gerrit, 10Patch-For-Review: [wm-checks-api] Create a new gerrit bot for Patch Demo - https://phabricator.wikimedia.org/T332474 (10hashar) Of course patchdemo requires a CORS header: ` Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://patchdemo.wmflabs.org/a... [14:38:37] I am seeing this on a gerrit change: [14:38:38] Error while fetching results for wm-patch-demo: TypeError: NetworkError when attempting to fetch resource. [14:38:45] never seen it before [14:38:55] "wm-patch-demo" or the network error [14:39:33] nor [14:41:54] 10Gerrit, 10Patch-For-Review: [wm-checks-api] Create a new gerrit bot for Patch Demo - https://phabricator.wikimedia.org/T332474 (10hashar) > Gerrit does not support a short URL having solely the change and patchset, it requires the repository name :-/ Turns out Gerrit does! The main page https://patchdemo.wm... [14:45:21] FYI, I'm going to remove doc1002 and doc2001 from getting updates, ahead of decommissioning them later this week: https://gerrit.wikimedia.org/r/c/operations/puppet/+/922872 (cc hashar , jnuche) [14:47:54] 👍 [14:48:08] sounds good to me, based on : doc.discovery.wmnet is an alias for doc1003.eqiad.wmnet. [14:48:35] and doc.wikimedia.org is up:) thanks for that eoghan [15:22:09] 10Release-Engineering-Team (They Live 🕶️🧟), 10Wikimedia-Phabricator-Extensions, 10serviceops-collab, 10Patch-For-Review: Disable "Browse Gerrit Projects" on https://phabricator.wikimedia.org/r/ - https://phabricator.wikimedia.org/T228507 (10Aklapper) [15:26:06] (03PS1) 10Hashar: wm-patch-demo: link to other patches [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/922882 (https://phabricator.wikimedia.org/T332474) [15:27:42] 10Gerrit, 10Patch-For-Review: [wm-checks-api] Create a new gerrit bot for Patch Demo - https://phabricator.wikimedia.org/T332474 (10hashar) With https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/922882: {F37031225 size=full} [15:27:48] (03CR) 10Hashar: "Example: https://phabricator.wikimedia.org/F37031225" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/922882 (https://phabricator.wikimedia.org/T332474) (owner: 10Hashar) [15:28:22] eoghan: +1 as well :] [15:28:57] I really dislike those OS upgrades, they are so prone to various bugs :/ [15:29:14] luckily there are a few pairs of eyes to handle them! [15:30:19] hashar: one thing I forgot, I think I broke a jenkins machine in cloud [15:30:45] "Puppet failure on jenkins-releases.devtools.eqiad1.wikimedia.cloud" [15:30:59] this looks like it was probably jenkins UID change [15:31:19] so let me check on the test instance [15:37:05] mutante: possibly cause the prod uid/gid are used which would not be possible on WMCS maybe [15:37:36] I think we had the issue for beta cluster and the mwdeploy user [15:38:27] mabye set the uid to undef when in the labs realm? [15:38:32] sounds kind of familiar indeed [15:39:02] I can try to find the work around that is used for existing stuff [15:39:15] might be easier though: [15:39:16] Error: /Stage[main]/Jenkins/User[jenkins]/uid: change from 498 to 924 failed: Could not set uid on user[jenkins]: Execution of '/usr/sbin/usermod -u 924 jenkins' returned 8: usermod: user jenkins is currently used by process 15881 [15:39:21] stops jenkins :) [15:39:39] that would happen in prod too [15:39:53] puppet run needs to happen while jenkins is down [15:40:26] stopping jenkins and running puppet again.. and ... no more puppet error [15:40:29] done, heh [15:40:51] well, the puppet error is fixed. but I should still do the chmod and stuff [15:46:49] hashar: all fixed. short story is, it might be surprising but the cloud VM behaves just like prod. it's NOT a problem to have that 924 UID there. it just needed all the steps done in prod too [15:47:37] which is all the steps you wrote, plus find /var/lib/jenkins -gid 498 -exec chown :jenkins {} \; find /tmp -gid 498 -exec chown :jenkins {} \; and started service again [15:48:10] twas 498:498 in this case.. that was the trap that it was also sometimes 499 or group 496 etc [15:48:20] well, no more :) [15:53:10] 10Gerrit, 10Patch-For-Review: [wm-checks-api] Create a new gerrit bot for Patch Demo - https://phabricator.wikimedia.org/T332474 (10Esanders) Neat! Will this only be under Checks > Info? If so, can we make it more visible/discoverable somehow? [15:53:34] 10Release-Engineering-Team (Radar), 10SRE, 10serviceops-collab: Redirect revisions from svn.wikimedia.org to https://static-codereview.wikimedia.org - https://phabricator.wikimedia.org/T119846 (10thcipriani) [15:54:23] 10Gerrit, 10VPS-project-Codesearch, 10VPS-project-Extdist, 10serviceops-collab: Move clients off of gerrit-replica.wikimedia.org back to gerrit.wikimedia.org - https://phabricator.wikimedia.org/T336710 (10Legoktm) This caused {T337263} on the codesearch side (I honestly wouldn't have caught that ahead of t... [15:54:35] 10Release-Engineering-Team (Seen), 10SRE, 10serviceops-collab: URL shortener subdomains for useful Wikimedia infrastructure - https://phabricator.wikimedia.org/T223319 (10thcipriani) [16:14:31] 10Phabricator, 10serviceops-collab, 10Patch-For-Review: Phabricator monthly email: Only list Differential usage statistics for created and updated patchsets - https://phabricator.wikimedia.org/T337382 (10Dzahn) [16:16:02] 10Phabricator, 10serviceops-collab, 10Patch-For-Review: Phabricator monthly email: Only list Differential usage statistics for created and updated patchsets - https://phabricator.wikimedia.org/T337382 (10Dzahn) I added sre-collab to this because reviewing/testing/deploying patches to these queries is a thing... [16:18:09] 10Release-Engineering-Team (They Live 🕶️🧟), 10Wikimedia-Phabricator-Extensions, 10serviceops-collab, 10Patch-For-Review: Disable "Browse Gerrit Projects" on https://phabricator.wikimedia.org/r/ - https://phabricator.wikimedia.org/T228507 (10Dzahn) I still have https://gerrit.wikimedia.org/r/c/operations/pu... [16:21:21] 10Gerrit, 10VPS-project-Codesearch, 10VPS-project-Extdist, 10serviceops-collab: Move clients off of gerrit-replica.wikimedia.org back to gerrit.wikimedia.org - https://phabricator.wikimedia.org/T336710 (10Dzahn) > I guess we don't need to switch back to gerrit-replica? The agreement so far was "don't swit... [16:22:46] 10Gerrit, 10Patch-For-Review: [wm-checks-api] Create a new gerrit bot for Patch Demo - https://phabricator.wikimedia.org/T332474 (10hashar) >>! In T332474#8877519, @Esanders wrote: > Neat! Will this only be under Checks > Info? If so, can we make it more visible/discoverable somehow? I think the infos can be... [16:25:48] 10Gerrit, 10VPS-project-Codesearch, 10VPS-project-Extdist, 10serviceops-collab: Move clients off of gerrit-replica.wikimedia.org back to gerrit.wikimedia.org - https://phabricator.wikimedia.org/T336710 (10Dzahn) P.S. it also seems possible in the future, with regular 6 month DC-switchover, we would conside... [16:30:55] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10SRE, 10serviceops, 10Test-Coverage: Add pcov PHP extension to wikimedia apt (and upgrade from 1.0.6-4+wmf1~buster1 to 1.0.11) so it can be used in Wikimedia CI - https://phabricator.wikimedia.org/T243847 (10Jdforrester-WMF) [16:31:33] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10SRE, 10serviceops, 10Test-Coverage: Add pcov PHP extension to wikimedia apt (and upgrade from 1.0.6-4+wmf1~buster1 to 1.0.11) so it can be used in Wikimedia CI - https://phabricator.wikimedia.org/T243847 (10Jdforrester-WMF) >>! In T243847#828... [16:33:15] 10Continuous-Integration-Infrastructure, 10serviceops-collab, 10Patch-For-Review: Migrate doc hosts to Bullseye - https://phabricator.wikimedia.org/T319477 (10Dzahn) >>! In T319477#8876245, @eoghan wrote: > The migration went smoothly. I'm going to remove the two buster hosts from service this afternoon, and... [16:35:30] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Work out how to bring in new Phabricator translations again - https://phabricator.wikimedia.org/T318763 (10brennen) [16:38:42] mutante: ah good catch. Thanks for the fix up! [16:42:49] 10Release-Engineering-Team, 10Data-Persistence (work done), 10Security-API-Service, 10Security-Team, and 2 others: Determine CI best practices for service which connects to MySQL - https://phabricator.wikimedia.org/T308789 (10thcipriani) @sbassett is using helm test as part of the pipeline still desirable... [16:43:20] hashar: yep:) will get back to the PHP upgrade tomorrow I think, after gerrit-replica [16:43:24] 10Release-Engineering-Team (Seen), 10Data-Persistence (work done), 10Security-API-Service, 10Security-Team, and 2 others: Determine CI best practices for service which connects to MySQL - https://phabricator.wikimedia.org/T308789 (10thcipriani) [16:46:38] 10Release-Engineering-Team (Radar), 10SRE, 10SRE-OnFire, 10serviceops-collab, 10Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162 (10thcipriani) [16:48:10] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Unable to preview video in Phabricator task comments and descriptions - https://phabricator.wikimedia.org/T309222 (10thcipriani) This will come after phorge migration [16:49:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Zuul: Zuul jenkins-bot user holding open SSH sessions - https://phabricator.wikimedia.org/T309376 (10thcipriani) p:05Triage→03Low [16:49:32] 10Release-Engineering-Team (Radar), 10SRE, 10SRE-OnFire, 10serviceops-collab, 10Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162 (10hashar) This was long forgotten. The problem is when a `Scap::Target` is removed from Puppet, it is not necessarily c... [16:52:54] 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops: PendingDeprecationWarning on update_version.py - https://phabricator.wikimedia.org/T310133 (10thcipriani) [16:54:53] 10Phabricator (Upstream), 10Release-Engineering-Team (Priority Backlog 📥), 10Upstream, 10User-brennen: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10thcipriani) p:05Medium→03Low Part of this workflow still needs fixing, but this wil... [16:56:04] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Data-Engineering-Planning, 10Performance Issue: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10thcipriani) [16:56:21] 10GitLab (CI & Job Runners), 10Data-Engineering-Planning, 10Performance Issue: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10thcipriani) [16:56:54] 10Release-Engineering-Team (Seen), 10Discovery-Search, 10Elasticsearch, 10Developer Productivity: Developer productivity: Shared ElasticSearch instance for local development environments, Quibble CI, and Patch Demo - https://phabricator.wikimedia.org/T312198 (10thcipriani) [17:00:23] 10Release-Engineering-Team, 10Scap, 10User-brennen: scap should have a better error message when it can't find keyholder key - https://phabricator.wikimedia.org/T313624 (10thcipriani) [17:00:48] 10Release-Engineering-Team (They Live 🕶️🧟), 10Scap, 10User-brennen: scap should have a better error message when it can't find keyholder key - https://phabricator.wikimedia.org/T313624 (10thcipriani) [17:01:51] 10Phabricator, 10Release-Engineering-Team: Make Taavi a Phab admin - https://phabricator.wikimedia.org/T337065 (10hashar) antoine-approve [17:02:25] 10Phabricator, 10Release-Engineering-Team: Make Taavi a Phab admin - https://phabricator.wikimedia.org/T337065 (10brennen) 05Open→03Resolved a:03brennen Done! [17:04:20] 10Release-Engineering-Team, 10Wikimedia-GitHub: stop syncing and delete labs/private repo from github - https://phabricator.wikimedia.org/T315925 (10thcipriani) 05Open→03Declined > Despite the name, the contents of this repo is publicly visible! I'm inclined to not worry too much about this. [17:07:59] 10Release-Engineering-Team, 10Wikimedia-GitHub: stop syncing and delete labs/private repo from github - https://phabricator.wikimedia.org/T315925 (10demon) 05Declined→03Resolved Easy fix is to just set the repo to private on Github (just done). Then nobody will see it and send us warnings, and Gerrit repli... [17:11:29] 10Release-Engineering-Team, 10Wikimedia-GitHub: stop syncing and delete labs/private repo from github - https://phabricator.wikimedia.org/T315925 (10Dzahn) :)) [17:15:08] 10GitLab (Infrastructure), 10serviceops-collab: Cookbook sre.gitlab.upgrade fails when another backup is in progress - https://phabricator.wikimedia.org/T336490 (10Jelto) 05Open→03Resolved I did some test in `--dry-run` and the new `fail_for_running_backup()` method works as expected. I'm closing this tas... [17:25:34] 10Release-Engineering-Team (Seen), 10Data-Persistence (work done), 10Security-API-Service, 10Security-Team, and 2 others: Determine CI best practices for service which connects to MySQL - https://phabricator.wikimedia.org/T308789 (10sbassett) Hey @thcipriani - There are some changes in-flight for this proj... [17:35:51] 10Gerrit, 10Patch-For-Review: [wm-checks-api] Create a new gerrit bot for Patch Demo - https://phabricator.wikimedia.org/T332474 (10Esanders) I guess I meant something immediately visible, without having to go to the "Checks" tab, so that less technical users might find it quickly, but maybe gerrit doesn't let... [18:36:16] (03PS2) 10Hashar: wm-patch-demo: link to other patches [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/922882 (https://phabricator.wikimedia.org/T332474) [19:02:59] 10Gerrit, 10Patch-For-Review: [wm-checks-api] Create a new gerrit bot for Patch Demo - https://phabricator.wikimedia.org/T332474 (10hashar) The little chips are collapsed when there are more than 3 elements (and up to 6 for ERRORS). If I change the category from INFO to WARNING and head to https://gerrit.wiki... [19:04:01] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab, 10Patch-For-Review: Provide mechanism to publish to doc.wikimedia.org from GitLab CI - https://phabricator.wikimedia.org/T336168 (10Jdforrester-WMF) [19:07:52] 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Release, 10Train Deployments, 10Wikimedia-production-error: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10demon) [19:12:28] <^demon> Train update: rolled back from group1 pending answer on ^ [19:13:02] <^demon> Sent email to ops & engineering lists [19:17:51] 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Release, 10Train Deployments, 10Wikimedia-production-error: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Jdforrester-WMF) Presumably fallout from 504c1a9faf6f62a629... [19:26:53] 10Continuous-Integration-Config, 10Release-Engineering-Team (Radar), 10SRE, 10serviceops, 10Test-Coverage: Add pcov PHP extension to wikimedia apt (and upgrade from 1.0.6-4+wmf1~buster1 to 1.0.11) so it can be used in Wikimedia CI - https://phabricator.wikimedia.org/T243847 (10hashar) [19:43:09] 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Ladsgroup) We can pull that patch in mwdebug but I can't reproduce the error/w... [19:49:23] 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Ladsgroup) There is no error showing up anymore. I think it was something with... [19:50:31] 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Ladsgroup) https://logstash.wikimedia.org/goto/1e68b2f14a3fecec8bc44366de679ffb [20:27:25] 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Jdforrester-WMF) >>! In T337427#8878206, @Ladsgroup wrote: > There is no error... [22:01:50] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10nshahquinn-wmf) >>! In T706#5957478, @Aklapper wrote: > @Zoranzoki21, @nshahquinn-wmf: Hi, I have added you both. (Usual default disclaimer: Please follow [guidelines](https://www.me... [22:38:29] 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Zabe)