[00:00:34] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:34] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:23:22] 10Release-Engineering-Team, 10MediaWiki-Docker, 10Regression: php-xdebug no longer works in MediaWiki-Docker - https://phabricator.wikimedia.org/T349266 (10Krinkle) [06:24:54] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/PDFEmbed] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/967046 [06:24:56] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/PDFEmbed] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/967046 (owner: 10QChris) [07:44:56] 10Phabricator: Check how H293 and H362 differ - https://phabricator.wikimedia.org/T349275 (10Aklapper) p:05Triage→03Low [07:45:41] 10Phabricator: Check how H293 and H362 differ - https://phabricator.wikimedia.org/T349275 (10hashar) [07:53:18] 10Fresh, 10Release-Engineering-Team: Fresh lacks an in container npm cache - https://phabricator.wikimedia.org/T349276 (10hashar) [08:18:46] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: User research - https://phabricator.wikimedia.org/T348854 (10KBach) [08:18:57] 10Release-Engineering-Team, 10Tech-Docs-Team, 10Documentation: Deployment pipeline (GitLab/Kokkuri/Blubber) documentation improvements - https://phabricator.wikimedia.org/T342317 (10KBach) [08:19:10] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: User research - https://phabricator.wikimedia.org/T348854 (10KBach) 05Open→03In progress Interviews in progress. [08:19:54] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Review deployment pipeline documentation - https://phabricator.wikimedia.org/T348855 (10KBach) 05Open→03In progress [08:19:56] 10Release-Engineering-Team, 10Tech-Docs-Team, 10Documentation: Deployment pipeline (GitLab/Kokkuri/Blubber) documentation improvements - https://phabricator.wikimedia.org/T342317 (10KBach) [09:22:38] 10GitLab, 10Infrastructure-Foundations, 10collaboration-services, 10serviceops, 10Patch-For-Review: Container image reports in debmonitor are broken - https://phabricator.wikimedia.org/T348876 (10Jelto) > debmonitor does not contain any images that are build by gitlab pipelines Adding GitLab tag. [09:27:02] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Check how H293 and H362 differ - https://phabricator.wikimedia.org/T349275 (10Aklapper) 05Open→03Resolved a:03Aklapper Disabled H362; they had thesame conditions and actions [09:51:06] 10GitLab (CI & Job Runners), 10collaboration-services, 10serviceops: Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491 (10Jelto) >>! In T304491#9242773, @MatthewVernon wrote: > Documentation has now been updated in the main namespace - https://wikitech.wikimedia.org/w... [10:35:19] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10Aklapper) [10:35:30] 10Phabricator, 10Release-Engineering-Team (Social Piranhas 🐟): Document scap3 deployment of Phabricator - https://phabricator.wikimedia.org/T313956 (10Aklapper) 05Open→03Resolved All and any insignificant knowledge I could find in forgotten dusty corners of my brain has been ruthlessly dumped into https://... [10:53:15] 10Release-Engineering-Team (Social Piranhas 🐟), 10Striker, 10Patch-For-Review: Striker-created Diffusion mirrors of GitLab repos are empty (due to master vs main branch name mismatch) - https://phabricator.wikimedia.org/T348131 (10Aklapper) [10:56:37] 10Release-Engineering-Team (Social Piranhas 🐟), 10Striker, 10Patch-For-Review: Striker-created Diffusion mirrors of GitLab repos are empty (due to master vs main branch name mismatch) - https://phabricator.wikimedia.org/T348131 (10taavi) > This is due to using the default branch name `main` in Wikimedia GitL... [11:05:37] 10Release-Engineering-Team (Social Piranhas 🐟), 10Striker, 10Patch-For-Review: Striker-created Diffusion mirrors of GitLab repos are empty (due to master vs main branch name mismatch) - https://phabricator.wikimedia.org/T348131 (10Aklapper) @taavi: I think the answer to your question is "default" but let me... [11:10:14] 10Beta-Cluster-Infrastructure: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients - https://phabricator.wikimedia.org/T343925 (10taavi) I looked a bit into this prompted by @kostajh on IRC. Essentially, the situation in beta is like this: * There is a beta-specific mail server, `deploy... [11:29:21] 10Gerrit: BatchUserRights: Internal error - https://phabricator.wikimedia.org/T349288 (10ChasFortna) [12:39:50] 10Release-Engineering-Team (Priority Backlog 📥), 10Quibble, 10wikimedia-irc-libera, 10Patch-For-Review: Phase out #wikimedia-quibble IRC channel in favor of #wikimedia-releng - https://phabricator.wikimedia.org/T346901 (10Jdforrester-WMF) 05Open→03Resolved [12:53:21] 10GitLab (Infrastructure), 10collaboration-services: Setup alerting for GitLab projects size limits - https://phabricator.wikimedia.org/T316553 (10Jelto) I also did not find a existing exporter which could be used. So we have to implement a custom exporter here. The metric should be available in the project A... [14:38:15] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Social Piranhas 🐟): docpub: allow publishing for failed pipelines - https://phabricator.wikimedia.org/T349309 (10jnuche) [14:51:21] (03CR) 10Ahmon Dancy: [C: 03+1] build: dedupe npm dependencies [performance/fresnel] - 10https://gerrit.wikimedia.org/r/967137 (owner: 10Hashar) [14:52:02] (03CR) 10Ahmon Dancy: [C: 03+1] build: Update jsdoc-to-markdown to 8.0.0 [performance/fresnel] - 10https://gerrit.wikimedia.org/r/966933 (owner: 10Hashar) [14:52:14] (03CR) 10Ahmon Dancy: [C: 03+1] build: Update Qunit from 2.15.0 to 2.20.0 [performance/fresnel] - 10https://gerrit.wikimedia.org/r/967139 (owner: 10Hashar) [14:58:52] 10GitLab (Infrastructure), 10collaboration-services: Setup alerting for GitLab projects size limits - https://phabricator.wikimedia.org/T316553 (10thcipriani) >>! In T316553#9264727, @Jelto wrote: > Beside the project size there will be more use-cases in the future which need custom metrics. So we most probabl... [15:51:24] 10Release-Engineering-Team, 10RESTBase, 10serviceops: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T349318 (10Jgiannelos) [15:56:16] 10Beta-Cluster-Infrastructure: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10Cparle) @hashar or @jijiki any idea who might be able help solve this? The flickr foundation are working on a tool to copy freely licensed image from Flickr to commons (called [[ https://commons.w... [16:07:28] Hey! We had an issue deploying restbase (T349318). It looks like its a scap related issue since i dont find any logs that show errors on the nodejs service level. Any idea what went wrong ? [16:07:29] T349318: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T349318 [16:16:43] 10Release-Engineering-Team, 10RESTBase, 10serviceops: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T349318 (10hnowlan) The server in question is still depooled and the rollback failed, so it is now running the newer version of the codebase. This could be a side-effect of T346354 maybe? [16:27:44] 10Release-Engineering-Team, 10RESTBase, 10serviceops: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T349318 (10Jgiannelos) p:05Triage→03High [16:30:42] 10Release-Engineering-Team, 10RESTBase, 10serviceops: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T349318 (10hnowlan) [16:43:45] 10Release-Engineering-Team, 10RESTBase, 10serviceops: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T349318 (10thcipriani) >>! In T349318#9265886, @hnowlan wrote: > The server in question is still depooled and the rollback failed, so it is now running the newer version of the codebase.... [16:48:56] 10Release-Engineering-Team, 10RESTBase, 10serviceops: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T349318 (10thcipriani) For the first issue, scap is right, "check_endpoints_restbase" is not on `restbase2015`: ` deploy-service@restbase2015:~$ ls -lh /etc/nagios/nrpe.d/ | grep restbas... [17:12:13] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for mabualruz - https://phabricator.wikimedia.org/T345186 (10NHillard-WMF) Hi everyone - Is there another deployment training coming up that Moh could participate in / would it be possible to set one up if not? [19:44:40] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for mabualruz - https://phabricator.wikimedia.org/T345186 (10thcipriani) >>! In T345186#9266131, @NHillard-WMF wrote: > Hi everyone - > Is there another deployment training coming up that Moh could participate in / would it b... [19:44:44] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for mabualruz - https://phabricator.wikimedia.org/T345186 (10thcipriani) p:05Triage→03Medium [20:01:57] 10Release-Engineering-Team, 10RESTBase, 10serviceops: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T349318 (10thcipriani) For the rollback problem, `scap` didn't find a `.done` symlink. Which makes sense, there isn't one: ` deploy-service@restbase2025:~$ ls -lhA /srv/deployment/restb... [21:41:59] 10Release-Engineering-Team (Social Piranhas 🐟), 10Release, 10Train Deployments: 1.42.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T348355 (10thcipriani) p:05Triage→03Medium a:03dancy [21:42:30] 10Release-Engineering-Team (Social Piranhas 🐟), 10Release, 10Train Deployments: 1.42.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T348356 (10thcipriani) p:05Triage→03Medium a:03dduvall