[00:11:39] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10Niharika) @sbassett My bad -- sounds like there has been a mix-up of threads here. I was talking about the next steps for getting MaxMi... [01:12:23] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10sbassett) >>! In T288844#7350579, @Niharika wrote: > @sbassett My bad -- sounds like there has been a mix-up of threads here. I was tal... [07:06:44] 10serviceops, 10MW-on-K8s, 10SRE, 10MW-1.37-notes (1.37.0-wmf.20; 2021-08-23): Make HTTP calls work within mediawiki on kubernetes - https://phabricator.wikimedia.org/T288848 (10Joe) I think it would be interesting to actually do what @TK-999 suggested and actually intercept all HTTP requests and route the... [08:41:54] https://performance.wikimedia.org/arclamp/svgs/daily/2021-09-09.excimer-k8s-wall.all.svgz [08:42:10] https://performance.wikimedia.org/arclamp/svgs/daily/2021-09-09.excimer-k8s.all.svgz [08:42:15] https://performance.wikimedia.org/arclamp/svgs/daily/2021-09-10.excimer-k8s-wall.api.svgz [08:42:21] https://performance.wikimedia.org/arclamp/svgs/daily/2021-09-10.excimer-k8s.api.svgz [08:42:35] ^ joe akosiaris [09:39:08] akosiaris: apache logs from mw on k8s: https://logstash.wikimedia.org/goto/fbe4f950509c2605456d06f9a5984f19 [10:07:31] I told you they were nicely hidden [10:18:03] majavah: sorry I was unaware :P [10:37:07] 10serviceops, 10MW-on-K8s, 10SRE, 10Release-Engineering-Team (Radar): The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Joe) >>! In T285232#7349608, @dancy wrote: > Still not working: https://foundation.wikimedia.org/static/current/sk... [10:39:05] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Benchmark performance of MediaWiki on k8s - https://phabricator.wikimedia.org/T280497 (10jijiki) Last round of urls, same configuration, with the addition of a couple more requests: [[https://gerrit.wikimedia.org/r/c/operat... [10:55:22] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Benchmark performance of MediaWiki on k8s - https://phabricator.wikimedia.org/T280497 (10jijiki) [11:29:55] 10serviceops, 10Lift-Wing, 10Kubernetes, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Discussion: dedicated directory in the deployment-chart repository for ML services - https://phabricator.wikimedia.org/T286791 (10elukey) Three code reviews are pending to implement the new `ml-services`... [12:06:52] 10serviceops, 10GitLab, 10Patch-For-Review: GitLab replica in codfw - https://phabricator.wikimedia.org/T285867 (10Jelto) @Arnoldokoth some thoughts on the wrong settings on the replica after restore: When we import the dump of production GitLab to the repica, the setting for `home_page_url` doesn't match t... [12:39:04] 10serviceops, 10SRE, 10Patch-For-Review, 10Platform Team Initiatives (Containerise Services): Migrate node-based services in production to node10 - https://phabricator.wikimedia.org/T210704 (10MoritzMuehlenhoff) [12:39:29] 10serviceops, 10SRE, 10Patch-For-Review, 10Platform Team Initiatives (Containerise Services): Migrate node-based services in production to node10 - https://phabricator.wikimedia.org/T210704 (10MoritzMuehlenhoff) [12:54:53] 10serviceops, 10MW-on-K8s, 10SRE, 10SRE Observability: Make logging work for mediawiki in k8s - https://phabricator.wikimedia.org/T288851 (10Joe) a:03Joe [13:10:02] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Implement POC for istio ingess - https://phabricator.wikimedia.org/T290966 (10JMeybohm) p:05Triage→03Medium [13:10:34] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Implement POC for istio ingress - https://phabricator.wikimedia.org/T290966 (10JMeybohm) [13:11:29] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Implement POC for istio ingress - https://phabricator.wikimedia.org/T290966 (10JMeybohm) [13:11:32] 10serviceops, 10MW-on-K8s, 10SRE: Create a gateway in kubernetes for the execution of our "lambdas" - https://phabricator.wikimedia.org/T261277 (10JMeybohm) [13:21:35] 10serviceops, 10MW-on-K8s, 10SRE, 10SRE Observability: Make logging work for mediawiki in k8s - https://phabricator.wikimedia.org/T288851 (10Joe) Using `fatal-error.php` I determined that at the moment logging to mwlog1001 works, while it seems that we're not able to log to logstash. I strongly suspect thi... [13:28:21] _joe_: fyi, all current train blockers are UBN! by definition [13:29:06] <_joe_> majavah: it makes little sense as it's not like this is a stop-the-world-or-everything's-on-fire bug, but if that's the convention, ok :) [13:29:33] <_joe_> but on the positive side, it allowed me to see we have a problem with php error handling in k8s :) [13:36:05] the definition is more about not disrupting the train and all workflows that depend on it [13:40:32] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: kube-apiserver need to reach webhooks running inside of the cluster - https://phabricator.wikimedia.org/T290967 (10JMeybohm) [13:46:01] _joe_: those log docs are almost empty [13:47:07] e.g, what's to work with in https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-syslog-2021.09.14?id=COCN5HsB9aenX4524O9E ? [13:47:32] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: kube-apiserver need to reach webhooks running inside of the cluster - https://phabricator.wikimedia.org/T290967 (10JMeybohm) [13:48:01] I had found those too, but I disgarded them as useless for now [13:48:11] <_joe_> those are the apache access logs [13:48:46] <_joe_> correctly structured into fields, thanks to you [13:59:24] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Benchmark performance of MediaWiki on k8s - https://phabricator.wikimedia.org/T280497 (10jijiki) [14:50:33] 10serviceops, 10MW-on-K8s, 10SRE, 10Release-Engineering-Team (Radar): The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10dancy) @Joe I propose decoupling the webserver and app images by eliminating all uses of the 'php' symlink in oper... [15:28:27] 10serviceops, 10SRE, 10wikidiff2, 10Community-Tech (CommTech-Sprint-9), 10Platform Team Workboards (Platform Engineering Reliability): Deploy wikidiff2 1.12.0 - https://phabricator.wikimedia.org/T285857 (10WDoranWMF) Hi @ldelench_wmf, this is scheduled as part of our ops work. It will require some review... [16:10:00] 10serviceops, 10MW-on-K8s, 10SRE, 10SRE Observability: Make logging work for mediawiki in k8s - https://phabricator.wikimedia.org/T288851 (10dancy) @Joe @Krinkle What's the reason php7-fatal-error.php is in /etc/php (via operations/puppet) and not in operations/mediawiki-config ? [17:09:53] 10serviceops, 10MW-on-K8s, 10SRE, 10SRE Observability: Make logging work for mediawiki in k8s - https://phabricator.wikimedia.org/T288851 (10Krinkle) @dancy TLDR: It could probably be moved, and I'll ramble a bit about what I currently understand, some of which you know already, and these may or may not be... [19:27:07] 10serviceops, 10MW-on-K8s, 10SRE, 10Release-Engineering-Team (Radar): The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10dancy) @Krinkle Timo, is what I propose above feasible? [19:51:19] 10serviceops, 10MW-on-K8s, 10SRE, 10Release-Engineering-Team (Radar): The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Krinkle) >>! In T285232#7350176, @dancy wrote: > Can someone give me an example of a curl command that exercises t... [23:00:02] rzl: I moved httpbb's hourly tests over to cumin1001/eqiad, but didn't file a task for automating yet in case you had other plans