[04:54:31] 10serviceops, 10MW-on-K8s, 10SRE: Add the puppet CA to the MediaWiki deployment - https://phabricator.wikimedia.org/T284417 (10Joe) [04:57:37] 10serviceops, 10MW-on-K8s, 10SRE: Add conditional to mediawiki-config for stuff running on kubernetes - https://phabricator.wikimedia.org/T284418 (10Joe) [04:57:59] 10serviceops, 10MW-on-K8s, 10SRE: Add the puppet CA to the MediaWiki deployment - https://phabricator.wikimedia.org/T284417 (10Joe) p:05Triage→03High [04:58:15] 10serviceops, 10MW-on-K8s, 10SRE: Add conditional to mediawiki-config for stuff running on kubernetes - https://phabricator.wikimedia.org/T284418 (10Joe) p:05Triage→03High [05:00:03] 10serviceops, 10MW-on-K8s, 10SRE: Add all redis and memcached backends to mw on k8s automatically - https://phabricator.wikimedia.org/T284420 (10Joe) [05:00:12] 10serviceops, 10MW-on-K8s, 10SRE: Add all redis and memcached backends to mw on k8s automatically - https://phabricator.wikimedia.org/T284420 (10Joe) p:05Triage→03High [05:02:09] 10serviceops, 10MW-on-K8s, 10SRE: Enable TLS termination on the mwdebug deployment. fix the service definition - https://phabricator.wikimedia.org/T284421 (10Joe) [05:05:01] 10serviceops, 10MW-on-K8s, 10SRE: Enable TLS termination on the mwdebug deployment. fix the service definition in the chart - https://phabricator.wikimedia.org/T284421 (10Joe) p:05Triage→03High [05:13:36] 10serviceops, 10MW-on-K8s, 10SRE: Add the puppet CA to the MediaWiki deployment - https://phabricator.wikimedia.org/T284417 (10Joe) a:03Joe [05:41:56] 10serviceops, 10observability, 10Patch-For-Review: Outdated link in "MediaWiki exceptions and fatals" alert - https://phabricator.wikimedia.org/T284301 (10jijiki) 05Open→03Resolved a:03jijiki [07:47:46] <_joe_> I could use a second pair of eyes on https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/698456/ [07:55:02] _joe_: when building out the mediawiki image, where does the list of php and other packages come from anyways? [07:56:13] <_joe_> apergos: it's complicated :D [07:56:47] I never could have guessed :-D lay it on me! [07:57:08] or, point me to "crap I should read", that will be fine [07:57:09] <_joe_> but basically, https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/refs/heads/master/images/php/ [07:57:35] <_joe_> so every php-fpm image has https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/refs/heads/master/images/php/7.2/cli/Dockerfile.template (from php-cli) [07:57:44] right [07:57:47] <_joe_> plus https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/refs/heads/master/images/php/7.2/fpm/Dockerfile.template [07:58:02] _joe_: linter catched you getting the key reference in the contigmap wrong [07:59:22] ok, with luck I can poke around in that tree and find anything else I might need, thanks a heap! [07:59:59] <_joe_> jayme: great :) [08:01:51] <_joe_> apergos: ask if you have doubts to clarify [08:02:02] <_joe_> but if you just want the list of installed packages, there's debmonitor! [08:03:20] <_joe_> https://debmonitor.wikimedia.org/images/docker-registry.wikimedia.org/php7.2-fpm:0.1.4 [08:03:37] <_joe_> I need to refresh those images :) [08:07:27] 10serviceops, 10SRE, 10docker-pkg: Refresh all images in production-images - https://phabricator.wikimedia.org/T284431 (10Joe) [08:07:40] 10serviceops, 10SRE, 10docker-pkg: Refresh all images in production-images - https://phabricator.wikimedia.org/T284431 (10Joe) p:05Triage→03Medium [08:30:14] well if I want the list of installed packages there's dpkg -l | grep php on some appserver too :-P but anyways these links are a great starting point, tyvm! [08:30:36] (sorry for the latency, I was doing a timed run of some slides and couldn't stop in the middle) [08:56:12] <_joe_> what is on one appserver and what's in k8s differ quite a bit [10:33:40] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Enable TLS termination on the mwdebug deployment. fix the service definition in the chart - https://phabricator.wikimedia.org/T284421 (10jijiki) I have enabled TLS on staging for now, which will use some default certs [11:11:51] I'm interested in hearing why that is sometime [12:51:24] 10serviceops, 10MW-on-K8s, 10SRE: Add conditional to mediawiki-config for stuff running on kubernetes - https://phabricator.wikimedia.org/T284418 (10Reedy) [14:00:58] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Radar): Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10hashar) [14:01:37] 10serviceops, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10hashar) [14:09:28] 10serviceops, 10SRE: Refactor memcached modules - https://phabricator.wikimedia.org/T284454 (10jijiki) [14:09:40] 10serviceops, 10SRE, 10User-jijiki: Refactor memcached modules - https://phabricator.wikimedia.org/T284454 (10jijiki) [14:45:21] 10serviceops, 10Deployments, 10Prod-Kubernetes: Evaluate cost/benefits of switching to using a programming language to define kubernetes resources - https://phabricator.wikimedia.org/T254739 (10hashar) [14:48:54] 10serviceops, 10Platform Engineering, 10Release Pipeline, 10SRE, and 5 others: Kask functional testing with Cassandra via the Deployment Pipeline - https://phabricator.wikimedia.org/T224041 (10hashar) [14:59:47] will be a moment late, sorry [15:01:30] 10serviceops, 10SRE, 10Patch-For-Review, 10User-jbond: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10JMeybohm) We have another bunch of #release-engineering-team images failing, probably all due to being jessie based: ` Jun 7 11:28:11 deneb docker-report-rel... [15:25:52] 10serviceops, 10SRE, 10Patch-For-Review, 10User-jbond: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10MoritzMuehlenhoff) Can we remove all jessie-related containers from the registry? [15:33:23] 10serviceops, 10SRE, 10Patch-For-Review, 10User-jbond: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10Jdforrester-WMF) >>! In T251918#7138799, @JMeybohm wrote: > We have another bunch of #release-engineering-team images failing, probably all due to being jessie... [15:41:41] _joe_: I remember we talked about that in the past: Did we decide that we want to remove old/unused images (as far as we can) from the registry? [15:41:45] see ^^ [15:46:33] 10serviceops, 10Release Pipeline, 10SRE, 10Release-Engineering-Team (Radar), and 2 others: Remove obsoleted docker images - https://phabricator.wikimedia.org/T242604 (10JMeybohm) [15:47:06] 10serviceops, 10SRE, 10Patch-For-Review, 10User-jbond: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10JMeybohm) >>! In T251918#7138881, @MoritzMuehlenhoff wrote: > Can we remove all jessie-related containers from the registry? Yeah, we "kind of" can: https://w... [15:52:04] <_joe_> jayme: the releng images are still used [15:53:10] _joe_: how do you know. If I got James_F right, they are not [16:00:21] The jessie ones shouldn't be. [16:00:34] As in, I purged them out of existence as much as we can control from our side. [16:05:09] <_joe_> ack [16:05:22] <_joe_> jayme: we might want to add a simple script to purge all labels of an image from the registry [16:05:24] We're still working on killing off stretch, sadly. [16:06:10] _joe_: I believe docker-registryctl can already do this [16:06:27] <_joe_> delete all tags/ [16:07:28] usage: docker-registryctl delete-tags [-h] [--force] IMAGE_GLOB [16:08:00] IMAGE_GLOB The name (including the registry url) of the image. The tags to remove can be indicated as a glob pattern, or not at all. [16:08:55] <_joe_> ok then :P [16:13:48] I'll delete them tomorrow then [16:25:39] <_joe_> jayme: I fixed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/698456 (I forgot to commit a file!) [16:25:49] <_joe_> ok for me to merge? I want to test if it works :) [16:26:25] <_joe_> oops sorry, pretty late :) [17:15:20] 10serviceops, 10SRE, 10Patch-For-Review: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301) - https://phabricator.wikimedia.org/T280203 (10wiki_willy) Hi @Dzahn - do you have an ETA on when we can start removing these from the racks? We have a few installs that are partially compl... [17:15:45] 10serviceops, 10SRE, 10User-jbond, 10User-jijiki: Refactor memcached modules - https://phabricator.wikimedia.org/T284454 (10jbond)