[02:39:07] 10serviceops, 10SRE, 10Patch-For-Review: Package php 7.4 for wikimedia production - https://phabricator.wikimedia.org/T293449 (10Jdforrester-WMF) [05:59:05] <_joe_> mutante: yes that needs to alert because it's how we keep code on mw-on-k8s in sync right now [07:16:26] 10serviceops, 10MW-on-K8s, 10SRE, 10SRE Observability, 10Patch-For-Review: Make logging work for mediawiki in k8s - https://phabricator.wikimedia.org/T288851 (10Joe) [08:14:38] <_joe_> jayme / akosiaris: do you see any special reason why the container for php-fpm should be called {{ .Release.Name }}-app instead than, say, a fixed value like "mediawiki-php7.2-fpm" ? [08:17:44] _joe_: IIRC I've asked something along those lines as well in the past and the answer was that the containers should include {{ .Release.Name }} to make it easier to filter on in kibana [08:18:05] <_joe_> jayme: but we have the release name in another filter now [08:18:30] <_joe_> and in this case, this is actually harmful to searchability in kibana for us. [08:18:30] _joe_: is always was (like in the pod name) [08:18:53] <_joe_> ok so the reason is kibana, I think I can fix it [08:19:28] at least that is what I remember...maybe I can find the original question/answer - let me check [08:20:29] personally I like having the container names static and the same for multiple releases - multiple charts even [08:21:42] <_joe_> heh not sure about the latter part (but I can be convinced otherwise) [08:22:01] <_joe_> (multiple charts with the same container name... makes sense for sidecars maybe) [08:22:10] +1 to that [08:22:38] and it might also make sense to call a container "app" rather than "{{ .Release.Name }}-app" [08:23:33] <_joe_> yeah about that, i was thinking something like chartname-app [08:24:44] works as well. but things like chart name and release name are always provided in metadata AIUI [08:24:57] hm..maybe not in metrics, though [08:26:04] unfortunately I can't find my original question on that topic [08:34:42] 10serviceops, 10MW-on-K8s, 10SRE: Evaluate istio as an ingress for production usage - https://phabricator.wikimedia.org/T287007 (10JMeybohm) 05Open→03Resolved [08:34:48] 10serviceops, 10MW-on-K8s, 10SRE: Create a gateway in kubernetes for the execution of our "lambdas" - https://phabricator.wikimedia.org/T261277 (10JMeybohm) [08:35:33] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: kube-apiserver need to reach webhooks running inside of the cluster - https://phabricator.wikimedia.org/T290967 (10JMeybohm) [08:36:17] <_joe_> so, all metrics coming from k8s have a "release" and a "deployment" tag [08:36:43] <_joe_> whcih contain all the information we actually need to reconstruct [08:37:16] ah, okay. Nice [08:40:13] <_joe_> context is - I'm trying to find an easy way to select all log lines from all mediawiki deployments coming from the php container [08:44:17] yeah, I figured. I think that's what I wanted to do as well when I raised this topic [08:49:01] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Implement POC for istio ingress - https://phabricator.wikimedia.org/T290966 (10JMeybohm) [09:10:16] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Implement POC for istio ingress - https://phabricator.wikimedia.org/T290966 (10JMeybohm) [09:34:19] 10serviceops, 10MW-on-K8s, 10SRE, 10SRE Observability, 10Patch-For-Review: Make logging work for mediawiki in k8s - https://phabricator.wikimedia.org/T288851 (10Joe) The php-fpm logs are output to stderr, which goes to logstash at the moment using the physical node rsyslog, but it's under a different se... [10:51:14] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review, and 2 others: The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Joe) 05Open→03Resolved [10:51:26] 10serviceops, 10MW-on-K8s, 10SRE: Make all httpbb tests pass on the mwdebug deployment. - https://phabricator.wikimedia.org/T285298 (10Joe) [11:28:00] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Make all httpbb tests pass on the mwdebug deployment. - https://phabricator.wikimedia.org/T285298 (10Joe) [11:29:01] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review, and 2 others: The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Joe) 05Resolved→03Open Sadly I found a problem with our current approach: any file under static/current that is... [12:37:41] 10serviceops, 10MW-on-K8s, 10SRE, 10SRE Observability, 10Patch-For-Review: Make logging work for mediawiki in k8s - https://phabricator.wikimedia.org/T288851 (10JMeybohm) >>! In T288851#7443633, @Joe wrote: > One possible solution to all of our problems would be: > * let php-fpm log to two files, both in... [13:01:46] I was checking parsoid eqiad cluster load on grafana and looks like wtp1026 and wtp1042 aren't receiving any traffic? [13:04:31] <_joe_> subbu: I guess they're depooled, although I don't know why. https://config-master.wikimedia.org/pybal/eqiad/parsoid-php says they are [13:05:41] ok. [13:05:49] <_joe_> https://sal.toolforge.org/log/zBP5SnwB8Fs0LHO5qarz seems the relevant log line for 1026 [13:06:06] <_joe_> I remember it was having repeated issues, but I don't remember the details [13:06:13] <_joe_> I'll look into phabricator later [13:07:33] sounds good. thanks. no rush. thought I would flag it when i noticed it. [13:07:48] <_joe_> definitely, thanks :) [13:21:55] 10serviceops, 10MW-on-K8s, 10SRE, 10SRE Observability, 10Patch-For-Review: Make logging work for mediawiki in k8s - https://phabricator.wikimedia.org/T288851 (10akosiaris) > We also have another problem: how to treat and collect php slow logs. Right now I'm sending them to stderr but that gets us a lot o... [13:30:07] 10serviceops, 10MW-on-K8s, 10SRE, 10SRE Observability, 10Patch-For-Review: Make logging work for mediawiki in k8s - https://phabricator.wikimedia.org/T288851 (10Joe) >>! In T288851#7444291, @akosiaris wrote: >> We also have another problem: how to treat and collect php slow logs. Right now I'm sending th... [13:36:14] 10serviceops, 10MW-on-K8s, 10SRE, 10SRE Observability, 10Patch-For-Review: Make logging work for mediawiki in k8s - https://phabricator.wikimedia.org/T288851 (10akosiaris) >>! In T288851#7443633, @Joe wrote: > The php-fpm logs are output to stderr, which goes to logstash at the moment using the physical... [13:37:52] 10serviceops, 10MW-on-K8s, 10SRE, 10SRE Observability, 10Patch-For-Review: Make logging work for mediawiki in k8s - https://phabricator.wikimedia.org/T288851 (10Joe) >>! In T288851#7444368, @akosiaris wrote: >>>! In T288851#7443633, @Joe wrote: >> The php-fpm logs are output to stderr, which goes to log... [13:39:38] 10serviceops, 10MW-on-K8s, 10SRE, 10SRE Observability, 10Patch-For-Review: Make logging work for mediawiki in k8s - https://phabricator.wikimedia.org/T288851 (10akosiaris) >>! In T288851#7444372, @Joe wrote: >>>! In T288851#7444368, @akosiaris wrote: >>>>! In T288851#7443633, @Joe wrote: >>> The php-fpm... [13:50:41] _joe_: wtp1026 is me [13:51:16] I was running tests yesterday and still do, but I don't know about wtp1042 [15:14:52] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Performance-Team (Radar): June 2021 Datacenter switchover - https://phabricator.wikimedia.org/T281515 (10Marostegui) [15:26:54] 10serviceops, 10PHP 7.2 support, 10Patch-For-Review: Drop PHP 7.2 support from MediaWiki master branch, once Wikimedia production is on 7.4 - https://phabricator.wikimedia.org/T261872 (10Reedy) [17:09:17] 10serviceops, 10SRE, 10Patch-For-Review: Package php 7.4 for wikimedia production - https://phabricator.wikimedia.org/T293449 (10Legoktm) a:03Legoktm [17:12:57] 10serviceops, 10SRE: Package php 7.4 for wikimedia production - https://phabricator.wikimedia.org/T293449 (10Legoktm) [18:24:43] 10serviceops, 10SRE: Package php 7.4 for wikimedia production - https://phabricator.wikimedia.org/T293449 (10Reedy) Are we still installing `php-mongodb`? I can't see it obviously in puppet... If it is/was still in use, I'm guessing it was potentially xhgui stuff from #performance-team. `php-tidy` probably ca... [18:27:15] 10serviceops, 10SRE: Package php 7.4 for wikimedia production - https://phabricator.wikimedia.org/T293449 (10Dzahn) >>! In T293449#7445553, @Reedy wrote: > Are we still installing `php-mongodb`? I can't see it obviously in puppet... If it is/was still in use, I'm guessing it was potentially xhgui stuff from #p... [18:31:25] 10serviceops, 10SRE: Package php 7.4 for wikimedia production - https://phabricator.wikimedia.org/T293449 (10Dzahn) >>! In T293449#7445553, @Reedy wrote: > `php-tidy` probably can be answered by the Parsing people, but I don't think we're still installing it (at least, explicitly) either based on puppet T2164... [18:32:21] 10serviceops, 10SRE: Package php 7.4 for wikimedia production - https://phabricator.wikimedia.org/T293449 (10Legoktm) I copied the list out of what is currently packaged in the php72 component: https://apt-browser.toolforge.org/buster-wikimedia/component/php72/ >>! In T293449#7445553, @Reedy wrote: > Are we s... [18:32:48] 10serviceops, 10SRE: Package php 7.4 for wikimedia production - https://phabricator.wikimedia.org/T293449 (10Legoktm) [18:41:08] 10serviceops, 10SRE, 10Wikimedia-production-error: PHP7 corruption reports in 2020-2021 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10Majavah) [19:05:43] 10serviceops, 10Shellbox, 10User-brennen, 10Wikimedia-production-error: Shellbox\ShellboxError: Shellbox server returned status code 503 - https://phabricator.wikimedia.org/T292663 (10Legoktm) I think these are real 503s generated by Shellbox, from the shellbox-timeline httpd logs: ` 2021-10-02T23:51:01 1... [19:29:54] 10serviceops, 10SRE, 10Wikimedia-production-error: PHP7 corruption reports in 2020-2021 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10Krinkle) [20:32:43] 10serviceops, 10SRE: Package php 7.4 for wikimedia production - https://phabricator.wikimedia.org/T293449 (10Legoktm) [21:18:30] 10serviceops: How to update Maxmind geoip databases in MediaWiki k8s images - https://phabricator.wikimedia.org/T293939 (10Dzahn) [21:19:10] 10serviceops, 10MW-on-K8s: How to update Maxmind geoip databases in MediaWiki k8s images - https://phabricator.wikimedia.org/T293939 (10Dzahn) [21:24:01] 10serviceops, 10MW-on-K8s: How to update Maxmind geoip databases in MediaWiki k8s images - https://phabricator.wikimedia.org/T293939 (10Dzahn) intervals: puppetmaster pulling from maxmind: it tries once daily (2 separate jobs using 2 licenses) appservers pulling from puppetmaster: every puppet run expected u... [21:54:50] 10serviceops, 10SRE: rename OTRS role/module/cumin aliases - https://phabricator.wikimedia.org/T293942 (10Dzahn) [21:55:04] 10serviceops, 10SRE: rename OTRS role/module/cumin aliases - https://phabricator.wikimedia.org/T293942 (10Dzahn) 05Open→03Stalled [21:59:11] 10serviceops, 10SRE: rename OTRS role/module/cumin aliases - https://phabricator.wikimedia.org/T293942 (10Dzahn) please don't upload patches, i want to use this as an example in a kind of workshop [22:10:09] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE, 10Patch-For-Review: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10Dzahn) The new database files are now rolled out to all production app and API servers (mediawiki::canary_appserv... [22:10:54] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE, 10Patch-For-Review: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10Dzahn) @phuedx I think for your purposes this should be solved now. On our side we have to discuss how to do this... [22:11:13] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE, 10Patch-For-Review: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10Dzahn) 05In progress→03Resolved [22:12:53] 10serviceops, 10MW-on-K8s: How to update Maxmind geoip databases in MediaWiki k8s images - https://phabricator.wikimedia.org/T293939 (10Dzahn) [22:12:58] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE, 10Patch-For-Review: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10Dzahn) [22:13:58] 10serviceops, 10MW-on-K8s: How to update Maxmind geoip databases in MediaWiki k8s images - https://phabricator.wikimedia.org/T293939 (10Dzahn) duplicate of T288375 ? [22:29:52] 10serviceops, 10SRE: Package php 7.4 for wikimedia production - https://phabricator.wikimedia.org/T293449 (10Legoktm) > It would be great if we managed to build the packages so that php 7.2 and php 7.4 can coexist on the same application server, like debian tries to do. For PHP itself and the core extensions,... [22:30:20] 10serviceops, 10MW-on-K8s: How to update Maxmind geoip databases in MediaWiki k8s images - https://phabricator.wikimedia.org/T293939 (10Dzahn) [22:30:24] 10serviceops, 10MW-on-K8s: IPInfo MediaWiki extension depends on presence of maxmind db in the container/host - https://phabricator.wikimedia.org/T288375 (10Dzahn) [22:30:52] 10serviceops, 10observability: Enable mediawiki appserver metrics for jobrunner hosts - https://phabricator.wikimedia.org/T293943 (10Krinkle) [22:30:58] 10serviceops, 10observability, 10Performance-Team (Radar): Enable mediawiki appserver metrics for jobrunner hosts - https://phabricator.wikimedia.org/T293943 (10Krinkle) [22:31:08] 10serviceops, 10observability, 10Performance-Team (Radar): Enable mediawiki appserver metrics for jobrunner hosts - https://phabricator.wikimedia.org/T293943 (10Krinkle) [22:31:14] 10serviceops, 10observability, 10Performance-Team (Radar): Enable mediawiki appserver metrics for jobrunner hosts - https://phabricator.wikimedia.org/T293943 (10Krinkle) [22:33:10] 10serviceops, 10MW-on-K8s: IPInfo MediaWiki extension depends on presence of maxmind db in the container/host - https://phabricator.wikimedia.org/T288375 (10Dzahn) >>! In T288375#7296763, @wkandek wrote: > I believe the current frequency of MediaWiki deployments (multiple times a week) is good enough for the m... [22:41:48] 10serviceops, 10MW-on-K8s: IPInfo MediaWiki extension depends on presence of maxmind db in the container/host - https://phabricator.wikimedia.org/T288375 (10Dzahn) >>! In T288375#7364842, @akosiaris wrote: > a quick `du` in /usr/share/GeoIP points out that most files are well over 1MB (only GeoLite.dat is smal... [23:00:04] 10serviceops, 10SRE: Package php 7.4 for wikimedia production - https://phabricator.wikimedia.org/T293449 (10Legoktm) [23:43:43] 10serviceops, 10SRE: Package php 7.4 for wikimedia production - https://phabricator.wikimedia.org/T293449 (10Legoktm) [23:57:07] 10serviceops, 10SRE: Package php 7.4 for wikimedia production - https://phabricator.wikimedia.org/T293449 (10Legoktm)