[02:30:05] <wikibugs>	 10serviceops, 10Performance-Team (Radar): Migrate WMF Production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736 (10tstarling)
[09:03:10] <elukey>	 hello folks
[09:03:40] <elukey>	 I filed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/747460 since the current lint step for deployment-charts seems broken (missing the helm2 binary in the helm linter image)
[09:03:55] <elukey>	 not sure if we want to completely remove the function or not, I don't have strong opinions
[09:04:05] <elukey>	 but I'd love to merge some other patches :D
[09:04:16] <elukey>	 jelto: --^ (if you are caffeinated and up)
[09:05:50] <jelto>	 elukey: thanks for the patch. I have https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/746864 prepared which should fix the issue. But it's still in review. I try to fix the CI issue asap
[09:11:28] <elukey>	 ah didn't see it!
[09:12:16] <_joe_>	 sorry why does the image miss helm2?
[09:12:29] <_joe_>	 if we did that before changing CI, that patch needs to be reverted
[09:13:51] <_joe_>	 jelto: your change LGTM
[09:14:14] <jelto>	 joe: sorry yes there is a catch22 with helm2 and the removal. I planned to merge both at the same time but the docker image was updated recently. I have https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/746864 prepared for that
[09:14:35] <_joe_>	 yeah I just merged it
[09:14:41] <_joe_>	 now we also need luca's change I think
[09:15:29] <_joe_>	 or better, we need to remove that function completely
[09:16:07] <elukey>	 I think that jelto's patch removed the helm_version() call in linting, it should work
[09:16:11] <jelto>	 yes the helm_version(chart) is not used anymore, so we should clean that up as well. I'll make sure it's properly cleaned up but CI should be unstuck for now
[09:16:32] <_joe_>	 elukey: yeah exactly 
[09:17:14] <elukey>	 perfect thanks a lot for the quick response folks :)
[09:17:46] <elukey>	 (coffee while we wait for CI to verify the patch)
[09:17:53] <jelto>	 sorry for the confusion, I forgot to mention in the docker image update, that it has a hard dependency on the rake file change
[09:19:00] <_joe_>	 np jelto :)
[09:19:20] <_joe_>	 we don't even offer t-shirts for breaking CI
[09:19:34] <_joe_>	 it would cost too much
[09:19:45] <_joe_>	 but praise for fixing it too, that we do :)
[09:20:14] <_joe_>	 (not sure if you've seen https://commons.wikimedia.org/wiki/File:Framed_%22I_BROKE_WIKIPEDIA..._THEN_I_FIXED_IT!%22_T-shirt.jpg)
[09:20:50] <jelto>	 I know about the shirt yes :P
[09:32:32] <wikibugs>	 10serviceops, 10SRE, 10User-Elukey: Test memsniff as possible replacement of memkeys - https://phabricator.wikimedia.org/T228970 (10fgiunchedi)
[09:49:40] <wikibugs>	 10serviceops, 10Maps, 10Product-Infrastructure-Team-Backlog, 10User-jijiki: Maps 2.0 roll-out plan - https://phabricator.wikimedia.org/T280767 (10Jgiannelos)
[12:48:20] <wikibugs>	 10serviceops, 10SRE, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10Ladsgroup) FWIW the increase in memory slop is back to wmf.9: https://grafana.wikimedia.org/d/000000607/cluste...
[13:00:52] <wikibugs>	 10serviceops, 10Maps, 10Product-Infrastructure-Team-Backlog, 10Patch-For-Review, 10User-jijiki: Maps 2.0 roll-out plan - https://phabricator.wikimedia.org/T280767 (10TheDJ) @Jgiannelos question.. if we set `wgKartographerDfltStyle` to osm-tegola... shouldn't we be looking at modifying `wgKartographerStyl...
[13:19:15] <wikibugs>	 10serviceops, 10MW-on-K8s, 10Release Pipeline: Pushes to docker-registry fail for images with compressed layers of size >1GB - https://phabricator.wikimedia.org/T288198 (10JMeybohm) 05Resolved→03Open AIUI from IRC backlog we had issues again @dancy  / @Legoktm    ` 10.64.48.17 - ci-restricted [14/Dec/202...
[13:19:27] <wikibugs>	 10serviceops, 10MW-on-K8s, 10Release Pipeline: Pushes to docker-registry fail for images with compressed layers of size >1GB - https://phabricator.wikimedia.org/T288198 (10JMeybohm) p:05High→03Medium
[13:40:02] <wikibugs>	 10serviceops, 10MW-on-K8s: On the kube-experimental mwdebug cluster, MediaWiki sees all edits as coming from localhost - https://phabricator.wikimedia.org/T297613 (10Joe) a:03Joe Enabling `mod_remoteip` did the trick. I will now add the configuration to the base image.
[14:23:13] <wikibugs>	 10serviceops, 10Maps, 10Product-Infrastructure-Team-Backlog, 10Patch-For-Review, 10User-jijiki: Maps 2.0 roll-out plan - https://phabricator.wikimedia.org/T280767 (10Jgiannelos) I think eventually when we finish rolling out tegola, osm-intl and osm will eventually point to tegola as a vector tile source...
[15:33:46] <wikibugs>	 10serviceops, 10SRE, 10ops-codfw: Installation issues on PowerEdge R440 Kafka main codfw servers with buster / firmware update needed - https://phabricator.wikimedia.org/T297422 (10elukey) @Papaul Hi! Any chance that we could work on this today/tomorrow?
[16:11:52] <elukey>	 hello folks, I am going to shutdown kafka-main2003 to allow Papaul to upgrade the firmware etc..
[16:27:45] <_joe_>	 elukey: can you wait ~ 10 minutes so I'm done and I can go afk?
[16:28:31] <elukey>	 ahem the host is already down :D
[17:08:54] <elukey>	 host still under upgrade (BIOS + NIC), I will make it boot after maintenance is completed to see if it works or not
[17:09:06] <elukey>	 kafka* is masked, so in theory I could launch a reimage
[17:23:13] <wikibugs>	 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10dduvall) The removal of tiller has broken PipelineLib's `deploy` functionality. For example, https://integration.wikimedia.org/ci/job/blubber-pipeline-rehearse/84/console  We'll need to...
[17:24:18] <wikibugs>	 10serviceops, 10SRE, 10ops-codfw: Installation issues on PowerEdge R440 Kafka main codfw servers with buster / firmware update needed - https://phabricator.wikimedia.org/T297422 (10Papaul) a:03Papaul
[17:28:39] <wikibugs>	 10serviceops, 10Release Pipeline, 10Release-Engineering-Team (Priority Backlog 📥): PipelineLib deploy is broken and needs refactoring to use helm3 - https://phabricator.wikimedia.org/T297809 (10dduvall)
[17:45:33] <wikibugs>	 10serviceops, 10Patch-For-Review: Upgrade kafka-main nodes to buster - https://phabricator.wikimedia.org/T296641 (10Papaul)
[17:46:02] <wikibugs>	 10serviceops, 10SRE, 10ops-codfw: Installation issues on PowerEdge R440 Kafka main codfw servers with buster / firmware update needed - https://phabricator.wikimedia.org/T297422 (10Papaul) 05Open→03Resolved This is complete
[18:02:45] <elukey>	 kafka-main2003 up, will try the reimage tomorrow:)
[20:27:03] <wikibugs>	 10serviceops, 10Release Pipeline, 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥): PipelineLib deploy is broken and needs refactoring to use helm3 - https://phabricator.wikimedia.org/T297809 (10Jdforrester-WMF) I guess [[https://codesearch.wmcloud.org/deployed/?q=deploy%3A&i=nope&files=...
[20:51:30] <wikibugs>	 10serviceops, 10Internet-Archive, 10InternetArchiveBot: Determine appropriate API request limits for InternetArchiveBot - https://phabricator.wikimedia.org/T296577 (10Cyberpower678) p:05Triage→03Medium
[22:08:16] <wikibugs>	 10serviceops, 10Release Pipeline, 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥): PipelineLib deploy is broken and needs refactoring to use helm3 - https://phabricator.wikimedia.org/T297809 (10jeena) It looks like jenkins needs permissions to create secrets in order to do a helm releas...
[23:58:14] <wikibugs>	 10serviceops, 10MW-on-K8s, 10MediaWiki-SettingsLoader, 10Continuous-Integration-Config, 10Patch-For-Review: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10Legoktm) On all physical hosts now:  ` legoktm@cumin1001:~$ sudo cumin A:all-mw 'php -m | grep yaml' 352 ho...