[02:39:15] 10serviceops: Deploy PHP patch for DOM replaceChild/removeChild performance - https://phabricator.wikimedia.org/T291052 (10tstarling) [02:39:38] 10serviceops: Deploy PHP patch for DOM replaceChild/removeChild performance - https://phabricator.wikimedia.org/T291052 (10tstarling) [04:47:45] 10serviceops, 10DBA, 10Toolhub, 10database-backups: Setup production database for Toolhub - https://phabricator.wikimedia.org/T271480 (10Marostegui) @bd808 were you able to find any timeline for this? thanks! [04:59:39] 10serviceops, 10DBA, 10Toolhub, 10database-backups: Setup production database for Toolhub - https://phabricator.wikimedia.org/T271480 (10bd808) >>! In T271480#7354322, @Marostegui wrote: > @bd808 were you able to find any timeline for this? thanks! I have a deployment into the staging cluster, but have no... [05:01:54] 10serviceops, 10DBA, 10Toolhub, 10database-backups: Setup production database for Toolhub - https://phabricator.wikimedia.org/T271480 (10Marostegui) Thanks for the update, if you need something from us let me know! [06:11:09] <_joe_> any takers for https://phabricator.wikimedia.org/T291052? I'm happy to assist in rebuilding the php packages [06:11:26] <_joe_> moritzm: would it be overly complex to add this patch to our set of patches? [06:25:30] hello folks, I tried to update the dnd disc's desired state after the switchover in [06:25:33] https://gerrit.wikimedia.org/r/c/operations/puppet/+/721244/1/modules/profile/files/configmaster/disc_desired_state.py [06:26:01] it needs a review, I wasn't 100% sure about some services like swift (checked the current status but it seems still pointing to codfw, so probably temporary?) [06:31:17] 10serviceops, 10MW-on-K8s, 10SRE, 10SRE Observability: Make logging work for mediawiki in k8s - https://phabricator.wikimedia.org/T288851 (10Joe) The only real reason why we've used puppet there was to inject the statsd address easily IIRC. @Krinkle we do already include a "params" file, I think we can ju... [07:19:41] _joe_: seems fine from a quick glance I'll take care of backporting and updating the debs, then someone from service ops can take of the rollout [07:28:25] <_joe_> moritzm: <3 [07:46:57] 10serviceops, 10MW-on-K8s, 10SRE, 10Release-Engineering-Team (Radar): The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Joe) @dancy I like your idea, even if I generally don't like using rewrite rules much. I'll try to bake a set of... [07:48:37] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: kube-apiserver need to reach webhooks running inside of the cluster - https://phabricator.wikimedia.org/T290967 (10JMeybohm) [08:59:52] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Implement POC for istio ingress - https://phabricator.wikimedia.org/T290966 (10JMeybohm) [10:09:00] 10serviceops, 10MW-on-K8s, 10SRE: Evaluate istio as an ingress for production usage - https://phabricator.wikimedia.org/T287007 (10akosiaris) [10:17:29] 10serviceops, 10MW-on-K8s, 10SRE: Evaluate istio as an ingress for production usage - https://phabricator.wikimedia.org/T287007 (10akosiaris) A few questions/points: * I see a bullet point `TLS certificates need to be placed in the namespace of istio-ingressgateway` and a comment by @joe above, but it doesn... [11:19:05] 10serviceops, 10MW-on-K8s, 10SRE: Evaluate istio as an ingress for production usage - https://phabricator.wikimedia.org/T287007 (10JMeybohm) >>! In T287007#7354949, @akosiaris wrote: > A few questions/points: > > * I see a bullet point `TLS certificates need to be placed in the namespace of istio-ingressgat... [13:32:37] 10serviceops: Deploy PHP patch for DOM replaceChild/removeChild performance - https://phabricator.wikimedia.org/T291052 (10MoritzMuehlenhoff) I've made an updated PHP 7.2 package with a 7.2 backport of https://github.com/php/php-src/commit/781e6b4d214012e9b9c0cf96a239cdf9f948da91 Not yet uploaded to apt.wikimed... [13:43:12] hello folks, I made a proposal to split tokens and secrets for helmfile in https://gerrit.wikimedia.org/r/c/operations/puppet/+/720048, lemme know your thoughts when you have a moment [13:43:17] :) [13:53:18] 10serviceops: Deploy PHP patch for DOM replaceChild/removeChild performance - https://phabricator.wikimedia.org/T291052 (10ssastry) Sure, upgrading scandium would be the best way. There is a test running there right now, but I can stop it if you want to upgrade it now. Otherwise, tomorrow morning UTC would work. [13:55:07] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Radar): The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Joe) As it stands, my new configuration would mean that we're going to issue a permanent red... [13:56:24] <_joe_> elukey: I'll try to take a look today [14:04:49] thanksss [14:05:17] there is probably another way to go, but I can't find a better idea, so I am all ears for suggestions in case :) [14:41:06] 10serviceops: Deploy PHP patch for DOM replaceChild/removeChild performance - https://phabricator.wikimedia.org/T291052 (10MoritzMuehlenhoff) Sure thing, I'll upgrade scandium tomorrow morning then. [14:42:32] 10serviceops, 10Anti-Harassment, 10IP Info, 10SRE: Update MaxMind GeoIP2 license key and product IDs for application servers - https://phabricator.wikimedia.org/T288844 (10wkandek) a:03wkandek Yes, we will take a look to see how the new database can be put on all appservers. [14:42:42] 10serviceops, 10SRE: Deploy PHP patch for DOM replaceChild/removeChild performance - https://phabricator.wikimedia.org/T291052 (10MoritzMuehlenhoff) [15:10:25] 10serviceops, 10SRE, 10Wikifeeds: wikifeeds in codfw seems failing health checks intermittently - https://phabricator.wikimedia.org/T290445 (10elukey) @akosiaris: me and @JMeybohm drafted https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-06_Wikifeeds, that should be in a reasonable good stat... [15:41:30] 10serviceops, 10SRE, 10wikidiff2, 10Community-Tech (CommTech-Sprint-9), 10Platform Team Workboards (Platform Engineering Reliability): Deploy wikidiff2 1.12.0 - https://phabricator.wikimedia.org/T285857 (10ldelench_wmf) Hi @WDoranWMF, thanks for the heads up! That timeline should work. We don't expect mo... [15:53:05] legoktm: ah thanks -- sorry for the noise, good to know cold cache is definitely the problem :) I'll come back around to it soon and play with the timeouts [16:25:03] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.0.0 - https://phabricator.wikimedia.org/T291095 (10dancy) [16:25:48] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.0.0 - https://phabricator.wikimedia.org/T291095 (10dancy) [16:48:02] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Radar): The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10mmodell) >>! In T285232#7355412, @Joe wrote: > at the end of the day it seems to me that we... [16:57:49] 10serviceops, 10SRE, 10Wikifeeds: wikifeeds in codfw seems failing health checks intermittently - https://phabricator.wikimedia.org/T290445 (10akosiaris) Adding notes in the task before I start moving things in to the incident doc itself. * Docs are at https://wikitech.wikimedia.org/wiki/Wikifeeds. That be... [17:05:21] 10serviceops, 10GitLab, 10Patch-For-Review: GitLab replica in codfw - https://phabricator.wikimedia.org/T285867 (10Arnoldokoth) @Jelto I've taken note of this and will add it to the script. Thank you. [17:27:21] 10serviceops, 10SRE, 10wikidiff2, 10Community-Tech (CommTech-Sprint-9), 10Platform Team Workboards (Platform Engineering Reliability): Deploy wikidiff2 1.12.0 - https://phabricator.wikimedia.org/T285857 (10NRodriguez) +1 to Lauren. @WDoranWMF of note, this does not need to be escalated if there are high... [18:13:38] 10serviceops, 10MW-on-K8s, 10SRE, 10MW-1.37-notes (1.37.0-wmf.20; 2021-08-23): Make HTTP calls work within mediawiki on kubernetes - https://phabricator.wikimedia.org/T288848 (10Legoktm) My overdue status update: * MediaWiki now has a $wgLocalHTTPProxy setting, which allows setting a HTTP proxy for all req... [19:31:31] 10serviceops, 10MW-on-K8s, 10SRE, 10MW-1.37-notes (1.37.0-wmf.20; 2021-08-23): Make HTTP calls work within mediawiki on kubernetes - https://phabricator.wikimedia.org/T288848 (10Joe) Using envoy as a proper proxy instead than a transparent one makes me uneasy a bit; it's not how we've used it, and I don't... [19:42:51] 10serviceops: Migrate WMF Production from PHP 7.2 to a newer version - https://phabricator.wikimedia.org/T271736 (10RhinosF1) [20:18:49] 10serviceops, 10MW-on-K8s, 10Performance-Team, 10SRE, 10Traffic: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10jijiki) [20:19:48] 10serviceops, 10MW-on-K8s, 10Performance-Team, 10SRE, 10Traffic: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10jijiki) [23:37:21] 10serviceops, 10MW-on-K8s, 10SRE, 10MW-1.37-notes (1.37.0-wmf.20; 2021-08-23): Make HTTP calls work within mediawiki on kubernetes - https://phabricator.wikimedia.org/T288848 (10Legoktm) We could teach MediaWiki how to use a transparent proxy instead, I'll poke at that.