[08:36:39] Hi, I just merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1175165 and then run puppet agent and tried to run deploy and it was empty [08:36:50] https://www.irccloud.com/pastebin/yycVcIGE/ [08:37:00] Am I doing something wildly wrong? [08:38:30] probably it's not needed [09:02:33] Amir1: checking [09:03:31] Wait it was already absented [09:03:38] So it was already removed from kubernetes [09:03:55] (if a deploy was ran after it was absented) [09:25:49] 06serviceops, 06SRE: deploy1003 running out of disk space - https://phabricator.wikimedia.org/T401647#11077261 (10Clement_Goubert) a:03Clement_Goubert [09:29:43] 06serviceops, 06SRE: deploy1003 running out of disk space - https://phabricator.wikimedia.org/T401647#11077281 (10Clement_Goubert) The `/srv/docker/overlay2` is taking up 177GB because we keep 7 days of images, which is probably way overkill. I'll run a prune keeping the last 3 days and will update the relevan... [09:31:43] 06serviceops, 06SRE: deploy1003 running out of disk space - https://phabricator.wikimedia.org/T401647#11077286 (10Clement_Goubert) ` # /usr/bin/docker image prune --all --force --filter until=72h [...] Total reclaimed space: 102.2GB ` [09:54:15] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11077334 (10MoritzMuehlenhoff) [10:28:52] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11077422 (10MoritzMuehlenhoff) a:05MoritzMuehlenhoff→03None >>! In T400637#11075140, @wiki_willy wrote: > Hi @MoritzMuehlenhoff - are you able to help confirm... [10:36:30] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11077435 (10MoritzMuehlenhoff) [10:42:18] 06serviceops, 06SRE, 13Patch-For-Review: deploy1003 running out of disk space - https://phabricator.wikimedia.org/T401647#11077453 (10Clement_Goubert) 05Open→03Resolved Old images will now be pruned every 3 days, and disk space is at manageable levels [10:55:47] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11077507 (10MoritzMuehlenhoff) a:05MoritzMuehlenhoff→03None >>! In T400638#11075133, @wiki_willy wrote: > Hi @MoritzMuehlenhoff - are you able to confirm the... [14:56:35] 06serviceops, 10DiscussionTools, 10MediaWiki-REST-API, 10RESTBase Sunsetting, and 3 others: Reroute RESTbase Endpoints for Linter Errors to MediaWiki REST Endpoints - https://phabricator.wikimedia.org/T384216#11078423 (10OWresch-WMF) [16:01:31] 06serviceops, 10Cassandra, 06Growth-Team, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#11078938 (10Scott_French) Following up, this is now scheduled for the [[ https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250812... [16:26:01] 06serviceops, 06MW-Interfaces-Team, 07OKR-Work, 13Patch-For-Review: Identify and configure rest.php routes in REST gateway - https://phabricator.wikimedia.org/T400132#11079078 (10BPirkle) >>! In T400132#11056855, @Clement_Goubert wrote: > #mw-interfaces-team This is the list of routes we've found in the sp... [16:38:03] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11079185 (10wiki_willy) Thanks @MoritzMuehlenhoff! [16:38:20] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11079186 (10wiki_willy) Awesome, thank you! [16:59:28] 06serviceops, 10Scap: Provide MediaWiki app image PHP version in helm values - https://phabricator.wikimedia.org/T401721 (10Scott_French) 03NEW [17:58:09] 06serviceops, 10Cassandra, 06Growth-Team, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#11079578 (10Scott_French) Alright, a bit more than 20m on from when the deployment completed, things are looking good. [[ https://grafana.wikime... [18:48:42] 06serviceops, 10Scap: Provide MediaWiki app image PHP version in helm values - https://phabricator.wikimedia.org/T401721#11079780 (10Scott_French) 05Open→03In progress p:05Triage→03Medium [18:51:14] 06serviceops, 10MW-on-K8s: Custom dblists for mwscript-k8s - https://phabricator.wikimedia.org/T401737 (10RLazarus) 03NEW [19:22:08] 06serviceops, 10Scap, 13Patch-For-Review: Provide MediaWiki app image PHP version in helm values - https://phabricator.wikimedia.org/T401721#11079929 (10Scott_French) I should mention here, there's one notable downside to using a metadata label like this: If for some reason the selected image no longer exist... [20:09:23] 06serviceops, 10Scap, 13Patch-For-Review: Provide MediaWiki app image PHP version in helm values - https://phabricator.wikimedia.org/T401721#11080142 (10dancy) >>! In T401721#11079929, @Scott_French wrote: > I should mention here, there's one notable downside to using a metadata label like this: If for some... [22:52:30] 06serviceops, 10Ceph, 06Data-Persistence, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#11080722 (10bd808) [22:52:36] 06serviceops, 13Patch-For-Review: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#11080721 (10bd808) [23:33:27] 06serviceops, 10Cassandra, 06Growth-Team, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#11080796 (10Eevans) [23:33:41] 06serviceops, 10Cassandra, 06Growth-Team, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#11080797 (10Eevans) [23:34:20] 06serviceops, 10Cassandra, 06Growth-Team, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#11080798 (10Eevans)