[00:53:54] 10Phabricator, 10VPS-project-Phabricator, 10serviceops-collab: https://phab.wmflabs.org/ is down (Can Not Connect to MySQL) - https://phabricator.wikimedia.org/T334801 (10EpicPupper) 05Resolvedβ†’03Open [00:55:07] 10Phabricator, 10VPS-project-Phabricator, 10serviceops-collab: https://phab.wmflabs.org/ is down (Can Not Connect to MySQL) - https://phabricator.wikimedia.org/T334801 (10EpicPupper) {F37096036} [08:21:39] 10Release-Engineering-Team, 10Patch-For-Review, 10Puppet: Puppet git::clone probably does not need `umask` parameter - https://phabricator.wikimedia.org/T338277 (10hashar) a:03hashar [08:38:31] jeena: are you running the train today? [08:39:01] How much time do we have to find a fix for T338264? [08:39:02] T338264: Caught exception of type Flow\Exception\DataModelException when trying to submit on MediaWiki.org - https://phabricator.wikimedia.org/T338264 [08:40:19] ...according to the deployment calendar, it looks like we have until 18:00 UTC. Would that be ok? [08:40:35] If we don't have anything by then, we'll revert the offending path [08:40:38] *patch [08:41:08] hashar: I guess jeena is still asleep, can you confirm? [08:41:09] I'll be doing the train during the USA time window [08:41:20] oh hey, you are awake :) [08:41:45] not for long :p [08:42:15] So anyway I think you are free to experiment at the moment [08:43:54] And yes just confirming that would be 1800 UTC [08:44:21] ok, thank you! We'll get right on it. [08:56:01] duesen: I am around :) [08:56:06] sorry was finishing up some patches [10:45:35] maintenance-disconnect-full-disks build 497938 integration-agent-docker-1036 (/: 29%, /srv: 10%, /var/lib/docker: 95%): OFFLINE due to disk space [10:55:33] maintenance-disconnect-full-disks build 497940 integration-agent-docker-1036 (/: 29%, /srv: 10%, /var/lib/docker: 95%): still OFFLINE due to disk space [10:57:35] 10Phabricator: When I delete a comment, I don't want to be added as a subscriber - https://phabricator.wikimedia.org/T338306 (10Reedy) [11:20:49] maintenance-disconnect-full-disks build 497945 integration-agent-docker-1036 (/: 29%, /srv: 10%, /var/lib/docker: 95%): still OFFLINE due to disk space [11:45:31] maintenance-disconnect-full-disks build 497950 integration-agent-docker-1036 (/: 29%, /srv: 10%, /var/lib/docker: 95%): still OFFLINE due to disk space [12:01:35] hmm who knows really [12:01:40] yzaiei2cl172 regular 13.9GB 3 hours ago 3 hours ago 1 false [12:01:52] from the Docker build cache [12:04:05] with some 9GB opt/lib/python/site-packages/torch [12:10:47] maintenance-disconnect-full-disks build 497955 integration-agent-docker-1036 (/: 29%, /srv: 10%, /var/lib/docker: 95%): still OFFLINE due to disk space [12:13:36] 10GitLab (Auth & Access), 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10CAS-SSO, 10Infrastructure-Foundations, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10Jelto) After fixing the `redirect_uri` I'm able to login successfully to the admin interface... [12:33:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Machine-Learning-Team: Python torch fills disk of CI Jenkins instances - https://phabricator.wikimedia.org/T338317 (10hashar) [12:33:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Machine-Learning-Team: Python torch fills disk of CI Jenkins instances - https://phabricator.wikimedia.org/T338317 (10hashar) Looking at the diff overlay which is at `/var/lib/docker/overlay2/yzaiei2cl172qsj37gazfomm2/diff` gives fun: 1... [12:35:34] maintenance-disconnect-full-disks build 497960 integration-agent-docker-1036 (/: 29%, /srv: 10%, /var/lib/docker: 95%): still OFFLINE due to disk space [12:43:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Machine-Learning-Team: Python torch fills disk of CI Jenkins instances - https://phabricator.wikimedia.org/T338317 (10hashar) The `Build Cache` is for Buildkit which is "hidden" from regular docker but actable on via `docker buildx`. Fro... [12:43:52] !log integration-agent-docker-1036: `docker buildx prune` to reclaim 21G of disk space # T338317 [12:43:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:43:54] T338317: Python torch fills disk of CI Jenkins instances - https://phabricator.wikimedia.org/T338317 [12:44:17] OKR: clear out disk space on machines :-\ [12:45:38] maintenance-disconnect-full-disks build 497962 integration-agent-docker-1036 (/: 29%, /srv: 10%, /var/lib/docker: 3%): RECOVERY disk space OK [12:56:51] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Machine-Learning-Team: Python torch fills disk of CI Jenkins instances - https://phabricator.wikimedia.org/T338317 (10isarantopoulos) Some info related to the above change: We have switched to this specific pytorch build (the one defined... [13:32:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Machine-Learning-Team: Python torch fills disk of CI Jenkins instances - https://phabricator.wikimedia.org/T338317 (10hashar) 05Openβ†’03Resolved a:03hashar I do not know how large the layer was before that change installing pytorch f... [13:43:26] (03Abandoned) 10Hashar: Zuul: [operations/grafana-grizzly] Add CI [integration/config] - 10https://gerrit.wikimedia.org/r/901598 (https://phabricator.wikimedia.org/T331659) (owner: 10Ayounsi) [14:14:57] (03Abandoned) 10Hashar: Publish coverage for Wikibase on post merge [integration/config] - 10https://gerrit.wikimedia.org/r/448560 (owner: 10WMDE-leszek) [14:15:33] (03Abandoned) 10Hashar: Zuul: [mediawiki/extensions/WikiLambda] Add JS coverage reporting [integration/config] - 10https://gerrit.wikimedia.org/r/677011 (owner: 10Jforrester) [14:16:36] 10Phabricator (Upstream), 10Upstream: When I delete a comment, I don't want to be added as a subscriber - https://phabricator.wikimedia.org/T338306 (10Aklapper) p:05Triageβ†’03Low [14:48:21] 10Phabricator, 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Patch-For-Review: Reports: RuntimeException due to wrong assumption that boards have more columns than default Backlog - https://phabricator.wikimedia.org/T336105 (10thcipriani) a:05Aklapperβ†’03brennen [14:49:32] 10Phabricator, 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Patch-For-Review: Reports: "Older" bar link in Histogram has wrong task query URL - https://phabricator.wikimedia.org/T336175 (10thcipriani) a:05Aklapperβ†’03brennen [14:49:55] 10Phabricator, 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Patch-For-Review: Reports: Age histogram shows nonsensical additional bucket after "Older" for projects created in last days - https://phabricator.wikimedia.org/T336152 (10thcipriani) a:05Aklapperβ†’03brennen [14:50:31] 10Phabricator, 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Patch-For-Review: InvalidArgumentException trying to render workboards when "Other Assignee" undefined - https://phabricator.wikimedia.org/T336135 (10thcipriani) a:05Aklapperβ†’03brennen [14:51:31] 10Phabricator, 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Patch-For-Review: Custom default project menu items have wrong description, typos, use outdated code repo URL - https://phabricator.wikimedia.org/T337297 (10thcipriani) a:05Aklapperβ†’03brennen [15:18:33] 10Phabricator, 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Patch-For-Review: Uninstall Phrequent (Phabricator application) - https://phabricator.wikimedia.org/T337606 (10Aklapper) [15:18:39] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Wikimedia-Phabricator-Extensions, 10Patch-For-Review: Improve error handling in "Escalate security issue" code - https://phabricator.wikimedia.org/T337654 (10Aklapper) [15:18:59] 10Phabricator, 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Patch-For-Review: Remove "Prototype" suffix from "Reports" menu item on Project pages - https://phabricator.wikimedia.org/T337876 (10Aklapper) [15:37:13] 10Phabricator: Add a "Thanks" token - https://phabricator.wikimedia.org/T338133 (10Aklapper) p:05Triageβ†’03Low > U+1F64F Person with folded hands - can indicate sorrow or regret, can also indicate pleading, praying, bowing, or thanking So I'm not convinced about this one. Anyone has a better idea (or why med... [15:52:51] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, 10Patch-For-Review: contint2002 service implementation tracking - https://phabricator.wikimedia.org/T324659 (10hashar) >>! In T324659#8904634, @Dzahn wrote: > @hashar This new machine is on buster. Somehow I thought we did bullseye from t... [16:32:59] James_F: the releng/maven-java8:1.0.1 you did on monday is not on the docker registry, that causes build failure for joal: Error response from daemon: manifest for docker-registry.wikimedia.org/releng/maven-java8:1.0.1 not found: manifest unknown: manifest unknown [16:33:02] I guess it failed to build? [16:33:55] hashar: Argh, did it? :-( [16:34:43] yeah he is filing a task about it [16:34:51] I am retriggering the fab deploy_docker [16:34:59] Oh. [16:35:01] Java8. [16:35:07] Those are still stretch-based, right? [16:35:12] yeah [16:35:14] And SRE killed stretch-backports. [16:35:14] and fail to build [16:35:18] So they can't ever build again. [16:35:24] ah perfect [16:35:27] * James_F sighs. [16:35:27] so they are legacy snapshot [16:35:32] :-( [16:35:41] I should stop cookie licking that task to drop the old Stretch image and actually commit to migrate them all [16:35:42] Sorry, I didn't look closely enough! [16:35:57] hashar: Just delete the jobs and see who complains, then make it their job. ;-) [16:36:03] https://phabricator.wikimedia.org/T338343 [16:36:14] na na we need thos eimages for sure [16:36:29] 10Project-Admins: Create project for Flex Diagrams extension - https://phabricator.wikimedia.org/T338157 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Requested public project #flex_diagrams has been created: https://phabricator.wikimedia.org/project/view/6597/ (In case you need to edit the project or proje... [16:36:54] Right. Boo. [16:37:27] 10Phabricator: Make sure anti-vandalism features are up to snuff - https://phabricator.wikimedia.org/T84 (10demon) [16:37:40] (03PS1) 10Jforrester: Revert "jjb: Update maven-java8-based jobs to images with new gerrit IP" [integration/config] - 10https://gerrit.wikimedia.org/r/928100 (https://phabricator.wikimedia.org/T338343) [16:38:06] hashar: ^^ should get the jobs back working, except for being pointed at the old gerrit IP and so probably failing that way instead. [16:39:02] 10Continuous-Integration-Config, 10Data Pipelines, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 B), and 2 others: Wikimedia-event-utilities jenkins build failure - https://phabricator.wikimedia.org/T338343 (10hashar) The java8 imiages are based on Debian Stretch and can no more be rebuild.... [16:39:06] awesome thanks [16:39:27] Can you deploy? [16:39:51] in team meeting but I will multitask :) [16:39:55] (03CR) 10Hashar: [C: 03+2] Revert "jjb: Update maven-java8-based jobs to images with new gerrit IP" [integration/config] - 10https://gerrit.wikimedia.org/r/928100 (https://phabricator.wikimedia.org/T338343) (owner: 10Jforrester) [16:39:58] Ta. [16:41:07] (03Merged) 10jenkins-bot: Revert "jjb: Update maven-java8-based jobs to images with new gerrit IP" [integration/config] - 10https://gerrit.wikimedia.org/r/928100 (https://phabricator.wikimedia.org/T338343) (owner: 10Jforrester) [16:41:33] !log jjb: reverting '*maven-release*' and '*java8*' Jenkins jobs to previous java8 docker images # T338343 [16:41:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:41:35] T338343: Wikimedia-event-utilities jenkins build failure - https://phabricator.wikimedia.org/T338343 [16:41:47] joal: should be good now [16:41:55] OH NO [16:42:06] I forgot to pull :D [16:42:10] :) [16:42:20] James_F: thank you for the config revert [16:43:05] 10Project-Admins: Create project for Flex Diagrams extension - https://phabricator.wikimedia.org/T338157 (10Yaron_Koren) Thank you! [16:46:44] James_F: I will revert the java8 docker images updated and I guess add a test to prevent modifications to images based on stretch [16:47:29] 10Phabricator: Make sure anti-vandalism features are up to snuff - https://phabricator.wikimedia.org/T84 (10Aklapper) [16:48:01] joal: so yeah I think it is good now [16:48:14] hashar: +1 [16:48:19] Thanks a milion hashar and James_F :) [16:48:27] Happy to help. [16:49:06] 10Continuous-Integration-Config, 10Data Pipelines, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 B), and 2 others: Wikimedia-event-utilities jenkins build failure - https://phabricator.wikimedia.org/T338343 (10hashar) The Jenkins jobs should now point back to the last good image and the buil... [16:49:08] joal: you can `recheck` or remove CR+2 do it again [16:49:22] Done ! [16:50:05] then watch the build progress, the job shouuld use the previous image [16:52:57] I confirm it worked :) Thnanks again! [16:54:17] (03PS1) 10Hashar: Revert "Dockerfiles: [maven-java8] Update gerrit.wikimedia.org IP" [integration/config] - 10https://gerrit.wikimedia.org/r/928102 (https://phabricator.wikimedia.org/T278203) [16:54:47] joal: James_F: congratulations for the fix! [16:55:39] (03CR) 10Hashar: [C: 03+2] "The Jenkins job got rolled back to the previous image version https://gerrit.wikimedia.org/r/c/integration/config/+/928100/" [integration/config] - 10https://gerrit.wikimedia.org/r/928102 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [16:56:56] (03Merged) 10jenkins-bot: Revert "Dockerfiles: [maven-java8] Update gerrit.wikimedia.org IP" [integration/config] - 10https://gerrit.wikimedia.org/r/928102 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [17:05:26] joal: don't forget to close the java8 image task https://phabricator.wikimedia.org/T338343 :) [17:06:00] sure hashar - not sure if you'd prever to do it, or if you prefer us to do it :) [17:06:03] We'll do! [17:06:19] yeah I am never sure who should close :] [17:06:25] but since I haven't verified, I will let you do it! [17:09:04] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, 10Security: Address issues within certain Gitlab CI security templates - https://phabricator.wikimedia.org/T338034 (10mmartorana) 05Openβ†’03In progress p:05Triageβ†’03Medium [17:20:19] 10Continuous-Integration-Config, 10Data Pipelines, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 B), and 2 others: Wikimedia-event-utilities jenkins build failure - https://phabricator.wikimedia.org/T338343 (10Snwachukwu) Thank you @hashar [17:21:13] 10Continuous-Integration-Config, 10Data Pipelines, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 B), and 2 others: Wikimedia-event-utilities jenkins build failure - https://phabricator.wikimedia.org/T338343 (10Snwachukwu) 05Openβ†’03Resolved [18:06:14] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T337526 (10jeena) [18:15:48] (03CR) 10Dzahn: "should the change to gerrit_ssh_host_key still be applied? just not the changelogs?" [integration/config] - 10https://gerrit.wikimedia.org/r/928102 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [18:16:21] (03CR) 10Dzahn: "the way it is there is now an IP in there that will never work again" [integration/config] - 10https://gerrit.wikimedia.org/r/928102 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [18:24:56] (03CR) 10Hashar: [C: 03+2] Revert "Dockerfiles: [maven-java8] Update gerrit.wikimedia.org IP" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/928102 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [19:08:17] taavi: can you delete https://phabricator.wikimedia.org/T336101#8911072 too? [19:08:40] yes once I've waited patiently enough [19:10:55] taavi: thanks, I think I've managed to clean everything else up [20:31:34] 10Phabricator, 10MediaWiki-extensions-CentralAuth, 10Stewards-and-global-tools: CentralAuth locks should logged linked users out of Phabricator - https://phabricator.wikimedia.org/T338384 (10AntiCompositeNumber) [20:42:58] 10Phabricator, 10MediaWiki-extensions-CentralAuth, 10Stewards-and-global-tools, 10WMF-General-or-Unknown: CentralAuth locks should logged linked users out of Phabricator - https://phabricator.wikimedia.org/T338384 (10Reedy) [20:43:55] 10Phabricator, 10MediaWiki-extensions-CentralAuth, 10Stewards-and-global-tools, 10WMF-General-or-Unknown: CentralAuth locks should logged linked users out of Phabricator - https://phabricator.wikimedia.org/T338384 (10Reedy) I suspect most of the code can be used verbatim (ideally some refactoring to reduce... [20:46:36] 10Phabricator, 10MediaWiki-extensions-CentralAuth, 10Stewards-and-global-tools, 10WMF-General-or-Unknown: CentralAuth locks should logged linked users out of Phabricator - https://phabricator.wikimedia.org/T338384 (10taavi) Yeah. And a check to only process users that have a `mediawikiwiki` should deal wit... [20:46:45] 10Phabricator, 10MediaWiki-extensions-CentralAuth, 10Stewards-and-global-tools, 10WMF-General-or-Unknown: CentralAuth locks should logged linked users out of Phabricator - https://phabricator.wikimedia.org/T338384 (10Reedy) ` [21:44:43] Reedy: https://phabricator.wikimedia.org/conduit/method/user.m... [20:48:04] 10Phabricator, 10MediaWiki-extensions-CentralAuth, 10Stewards-and-global-tools, 10WMF-General-or-Unknown: CentralAuth locks should logged linked users out of Phabricator - https://phabricator.wikimedia.org/T338384 (10bd808) https://phabricator.wikimedia.org/conduit/method/user.mediawikiquery/ is the API to... [20:50:22] 10Project-Admins: Create project tag for Tech Docs Team - https://phabricator.wikimedia.org/T338387 (10apaskulin) [21:03:32] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for dvrandecic - https://phabricator.wikimedia.org/T338389 (10DVrandecic) [21:18:22] 10Phabricator, 10MediaWiki-extensions-CentralAuth, 10Stewards-and-global-tools, 10WMF-General-or-Unknown: CentralAuth locks should logged linked users out of Phabricator - https://phabricator.wikimedia.org/T338384 (10bd808) {T222209} could also be addressed when implementing this. Code could be extracted f... [21:40:31] 10Beta-Cluster-Infrastructure, 10VisualEditor, 10VisualEditor-VisualDiffs: Visual diff of page creation fails with β€œInvalid response from server” - https://phabricator.wikimedia.org/T338388 (10matmarex) This revision seems to suffer from some problem unrelated to visual diffs. If you try to open it in visual... [22:38:17] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Temporarily replace the Phabricator logo for Pride Month - https://phabricator.wikimedia.org/T337964 (10EpicPupper) I'm wondering if this is an example of unintentional logo misuse? The [[https://meta.wikimedia.org/wiki/Brand/logo#Logo_misuse|WMF bra... [23:16:30] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T337526 (10TheresNoTime)