[08:28:17] 10serviceops, 10Wikimedia-Apache-configuration: Investigate restricting match pattern on /wiki RewriteRule - https://phabricator.wikimedia.org/T357595 (10Joe) >>! In T357595#9544806, @RLazarus wrote: >> the intention was probably for this to match something a bit more restrictive (e.g., matching ^/wiki(/.*)?$)... [11:50:02] claime: I think you can merge / build the python-build images. I have tested all the changes :) [12:00:42] hashar: merged, launching build [12:02:06] hashar: it's gonna take a while, the spark and flink images have to be rebuilt as well and they are *long* [12:02:25] oh nooo [12:02:35] anyway, thanks for the merge. I am idling for lunch :) [12:22:12] hashar: well, they fail to build because of a sha issue, so they got skipped, your new python*-build images are ready [12:24:23] claime: I'd like to merge https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/997537 but I haven't merged anything in that repo before. is it as simple as https://wikitech.wikimedia.org/wiki/Kubernetes/Images#Production_images says or is there something else I need to do? [12:25:30] taavi: Nope, simple as. As I said before, you'll get failures for the openjdk-8-jre, spark, and flint images, I have to tell the owners there's an issue with the key check, but your image should build fine [12:25:46] You need to Verify +2 and Code-Review +2 because no CI [12:28:53] great, thanks [12:41:06] claime: awesome thank you so much [12:46:51] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1349.eqiad.wmnet with OS bullseye [12:46:58] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1367.eqiad.wmnet with OS bullseye [12:47:30] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1476.eqiad.wmnet with OS bullseye [12:47:34] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1477.eqiad.wmnet with OS bullseye [12:58:59] claime: I have confirmed the new images to work for MY use case :) thanks! [13:00:56] hashar: I sure hope they work for others >:] [13:01:11] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1477.eqiad.wmnet with OS bullseye executed with errors: - mw... [13:05:30] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1477.eqiad.wmnet with OS bullseye [13:16:11] 10serviceops, 10API Platform, 10CirrusSearch, 10MediaWiki-Configuration, and 2 others: Provide a method for internal services to run api requests for private wikis - https://phabricator.wikimedia.org/T345185 (10Gehel) [13:21:08] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1476.eqiad.wmnet with OS bullseye completed: - mw1476 (**PAS... [13:23:47] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1367.eqiad.wmnet with OS bullseye completed: - mw1367 (**PAS... [13:26:36] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1349.eqiad.wmnet with OS bullseye completed: - mw1349 (**PAS... [13:37:56] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1477.eqiad.wmnet with OS bullseye completed: - mw1477 (**PASS**) - Removed from... [14:20:16] 10serviceops, 10SRE: Container Image policy for non-k8s uses - https://phabricator.wikimedia.org/T357441 (10akosiaris) 05Open→03Resolved a:03akosiaris Thanks for posting the question and I hope I managed to help. I 'll resolve this in the interest of not having it linger open (we got enough open lingeri... [14:55:37] 10serviceops, 10Observability-Logging, 10iPoid-Service: Logs from ipoid-production-daily-updates-28465260-qbbz9 are not visible in Logstash after Feb 14, 2024 @ 23:40:08.183 - https://phabricator.wikimedia.org/T357616 (10kostajh) The long running cronjob had a timeout when connecting to MySQL. The logs from... [14:55:52] 10serviceops, 10Observability-Logging, 10iPoid-Service: Logs from ipoid-production-daily-updates-28465260-qbbz9 are not visible in Logstash after Feb 14, 2024 @ 23:40:08.183 - https://phabricator.wikimedia.org/T357616 (10kostajh) @colewhite @JMeybohm @Clement_Goubert I think we could mark this resolved, unl... [14:57:46] 10serviceops, 10Release Pipeline, 10SRE, 10Release-Engineering-Team (Radar), and 2 others: Remove obsoleted docker images - https://phabricator.wikimedia.org/T242604 (10Clement_Goubert) Following [[ https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/JPWXM7RHTGXN2YZALPOH3AOISA... [15:23:09] 10serviceops, 10Data-Engineering, 10MW-on-K8s, 10SRE: Migrate mw-page-content-change-enrich to mw-api-int - https://phabricator.wikimedia.org/T357785 (10Clement_Goubert) [15:24:12] 10serviceops, 10Prod-Kubernetes: PodSecurityPolicies will be deprecated with Kubernetes 1.21 - https://phabricator.wikimedia.org/T273507 (10JMeybohm) I'm not super convinced of 4...writing ValidationAdmissionPolicies is quite complex and there are so many corner cases. I tried implementing the first restrictio... [15:35:38] 10serviceops, 10Data-Engineering, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Migrate mw-page-content-change-enrich to mw-api-int - https://phabricator.wikimedia.org/T357785 (10Clement_Goubert) p:05Triage→03Medium [16:15:55] 10serviceops, 10EventStreams, 10Prod-Kubernetes, 10Data-Engineering (Sprint 9), and 2 others: eventstreams regularly uses more than 95% of its memory limit - https://phabricator.wikimedia.org/T357005 (10lbowmaker) [16:31:46] 10serviceops, 10Add-Link, 10Growth-Team, 10Prod-Kubernetes, and 2 others: linkrecommendation-internal regularly uses more than 95% of its memory limit - https://phabricator.wikimedia.org/T357122 (10Urbanecm_WMF) >>! In T357122#9528660, @akosiaris wrote: > Already bumped by 200Mi in a9f958e50e5f5f4a8 ( T266... [19:31:55] Hi, rough estimate: How many replicas mw-on-k8s is going to have assuming 100% [19:54:22] cdanis: since you're online :D ^ [19:54:49] why do you think I would know that 😂 [19:56:01] you know everything :D [19:59:24] do you mean how many pod replicas? [19:59:33] yeah [20:05:41] I think I can get a rough estimate with this [20:05:42] https://phabricator.wikimedia.org/T351074 [20:06:02] 460 basicallu [20:06:15] that won't account for the pre-existing baseline [20:06:25] Amir1: https://grafana.wikimedia.org/goto/jx7PA9hIz?orgId=1 [20:06:47] number of pods in any mediawiki namespace on k8s, per site [20:13:17] ah thanks [20:25:12] so i would guess somewhere in the neighborhood of 800-1000 [20:25:14] per site [20:29:41] thanks