[09:14:15] 10serviceops, 10Gerrit, 10Release-Engineering-Team (Seen): Rename operations/debs/poolcounter-prometheus-exporter to match other Prometheus repositories - https://phabricator.wikimedia.org/T239688 (10hashar) [10:22:18] 10serviceops, 10Maps, 10Patch-For-Review: Repool maps primaries in Kartotherian - https://phabricator.wikimedia.org/T355892 (10hnowlan) >>! In T355892#9515591, @Jgiannelos wrote: > This change looks like is causing an issue. From apps team: > ` > We seem to be getting intermittent 404s for certain urls, e.g.... [11:25:17] 10serviceops, 10Maps, 10Patch-For-Review: Repool maps primaries in Kartotherian - https://phabricator.wikimedia.org/T355892 (10Jgiannelos) For reference here is the ticket tracking the actual problem with maps1009 T356756 [12:02:51] 10serviceops, 10Infrastructure-Foundations, 10Patch-For-Review: Debian installer waits for input for network config during host reimage - https://phabricator.wikimedia.org/T356709 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by volans@cumin1002 for host sretest1001.eqiad.wmnet with... [12:03:31] 10serviceops, 10Infrastructure-Foundations, 10Patch-For-Review: Debian installer waits for input for network config during host reimage - https://phabricator.wikimedia.org/T356709 (10Volans) p:05Triage→03High a:03Volans [12:10:53] 10serviceops, 10Infrastructure-Foundations, 10Patch-For-Review: Debian installer waits for input for network config during host reimage - https://phabricator.wikimedia.org/T356709 (10Volans) Ok the sretest1001 reimage is going through, I'll leave the task open until the reimage finishes. The issue was a typo... [12:11:40] 10serviceops, 10Infrastructure-Foundations, 10Patch-For-Review: Debian installer waits for input for network config during host reimage - https://phabricator.wikimedia.org/T356709 (10Marostegui) es1029 is also going thru fine! [12:18:11] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw1386.eqiad.wmnet with OS bullseye [12:23:51] 10serviceops, 10Infrastructure-Foundations, 10Patch-For-Review: Debian installer waits for input for network config during host reimage - https://phabricator.wikimedia.org/T356709 (10kamila) mw1386 seems to be going fine now, so yes, we can close this. Sorry and thanks for finding the cause @Volans <3 [12:26:15] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw1388.eqiad.wmnet with OS bullseye [12:28:48] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw1390.eqiad.wmnet with OS bullseye [12:37:43] 10serviceops, 10Infrastructure-Foundations: Debian installer waits for input for network config during host reimage - https://phabricator.wikimedia.org/T356709 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by volans@cumin1002 for host sretest1001.eqiad.wmnet with OS bullseye completed: -... [12:40:27] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw1392.eqiad.wmnet with OS bullseye [12:41:59] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw1394.eqiad.wmnet with OS bullseye [12:42:41] 10serviceops, 10Infrastructure-Foundations: Debian installer waits for input for network config during host reimage - https://phabricator.wikimedia.org/T356709 (10Volans) 05Open→03Resolved [12:45:11] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw1396.eqiad.wmnet with OS bullseye [12:45:49] 10serviceops, 10iPoid-Service (iPoid 1.0): Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service - https://phabricator.wikimedia.org/T356766 (10kostajh) [12:46:37] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw1408.eqiad.wmnet with OS bullseye [12:49:23] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw2317.codfw.wmnet with OS bullseye [12:51:58] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw1386.eqiad.wmnet with OS bullseye completed: - mw1386 (**PASS**) - Removed from P... [12:52:19] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw2318.codfw.wmnet with OS bullseye [12:54:51] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw2319.codfw.wmnet with OS bullseye [13:03:01] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw2350.codfw.wmnet with OS bullseye [13:04:05] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw1390.eqiad.wmnet with OS bullseye completed: - mw1390 (**PASS**) - Removed from P... [13:04:19] 10serviceops, 10Infrastructure-Foundations: Debian installer waits for input for network config during host reimage - https://phabricator.wikimedia.org/T356709 (10Volans) For posterity I'd also like to mention how misleading was the error message, as the debian-installer UI looked like it was failing to get th... [13:05:50] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw2352.codfw.wmnet with OS bullseye [13:06:29] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw1388.eqiad.wmnet with OS bullseye completed: - mw1388 (**WARN**) - Removed from P... [13:07:49] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw2354.codfw.wmnet with OS bullseye [13:11:45] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw2356.codfw.wmnet with OS bullseye [13:15:32] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw1392.eqiad.wmnet with OS bullseye completed: - mw1392 (**PASS**) - Removed from P... [13:16:38] 10serviceops, 10Maps: Repool maps primaries in Kartotherian - https://phabricator.wikimedia.org/T355892 (10hnowlan) 05Resolved→03Open [13:17:49] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw1394.eqiad.wmnet with OS bullseye completed: - mw1394 (**PASS**) - Removed from P... [13:20:27] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw1396.eqiad.wmnet with OS bullseye completed: - mw1396 (**PASS**) - Removed from P... [13:23:01] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw1408.eqiad.wmnet with OS bullseye completed: - mw1408 (**PASS**) - Removed from P... [13:28:07] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw2317.codfw.wmnet with OS bullseye completed: - mw2317 (**PASS**) - Removed from P... [13:30:35] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw2318.codfw.wmnet with OS bullseye completed: - mw2318 (**PASS**) - Downtimed on I... [13:33:03] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw2319.codfw.wmnet with OS bullseye completed: - mw2319 (**PASS**) - Downtimed on I... [13:42:06] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw2350.codfw.wmnet with OS bullseye completed: - mw2350 (**PASS**) - Downtimed on I... [13:45:42] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw2352.codfw.wmnet with OS bullseye completed: - mw2352 (**PASS**) - Downtimed on I... [13:47:36] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw2354.codfw.wmnet with OS bullseye completed: - mw2354 (**PASS**) - Downtimed on I... [13:50:20] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw2356.codfw.wmnet with OS bullseye completed: - mw2356 (**PASS**) - Downtimed on I... [14:17:52] 10serviceops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [14:29:12] 10serviceops, 10CirrusSearch, 10Data-Platform-SRE, 10Discovery-Search: Requesting permission to enable kafka log compaction for page_rerender on kafka-main - https://phabricator.wikimedia.org/T354794 (10Gehel) Moving to our backlog board, to be picked up again after March 20th 2024 [15:11:14] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: prometheus-apache-exporter in buster does not support -log.format json - https://phabricator.wikimedia.org/T283861 (10Clement_Goubert) Image has been rebuilt with the new versions that support the json log format, CR incoming to bump the version accross the fleet. [16:09:47] 10serviceops, 10Maps: Repool maps primaries in Kartotherian - https://phabricator.wikimedia.org/T355892 (10Jgiannelos) maps1009,2009 should be ready to be repooled [16:41:19] Hey! We are working on disabling storage on Parsoid/RESTBase. From what I see in puppet restbase connects to parsoid via `parsoid-async` listener. Do we have a diagram that we can check how much the traffic increased ? [16:45:02] nemo-yiannis: https://grafana.wikimedia.org/d/VTCkm29Wz/envoy-telemetry?orgId=1&var-datasource=codfw%20prometheus%2Fops&var-origin=restbase&var-origin_instance=All&var-destination=parsoid-php ? [16:45:13] thanks akosiaris [16:45:51] not sure it is what you want btw, but that should indeed be the traffic from restbase to parsoid. [16:46:23] yeah thats what I was looking for [16:46:26] but not via the parsoid-async listener ... [16:46:37] yeah, that ^ [16:46:55] funnily, I don't see a parsoid-async listener in the dropdown [16:47:10] which might mean no traffic? [16:47:11] hm, https://gerrit.wikimedia.org/g/operations/puppet/+/04c081733e1beed7c5887ae237bb3c4a6f4d1979/modules/profile/manifests/restbase.pp#87 [16:47:18] which means it's not used [16:47:33] isn't this what is used as parsoid_uri ? [16:47:53] oh default value [16:48:54] config.yaml on a restbase host says 6502/tcp which is indeed parsoid-php [16:49:31] ok [16:49:32] so that's the graph you want I 'd say, parsoid-async is unused apparently [16:49:50] 👍 [16:50:23] ah, it's the name of the entry in the service proxy but the service that powers the entry in the service proxy is indeed parsoid-php [16:50:54] there is parsoid-php too ofc, which is powered by parsoid-php (surprise!) [16:51:06] cc duesen ^ [16:52:54] akosiaris: wmflib::service::get_url should still return the url with the port of the service proxy listener AIUI [16:53:19] or I'm reading it wrong [16:55:03] jayme: not sure I get what you mean... [16:55:13] 10serviceops, 10iPoid-Service (iPoid 1.0): Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service - https://phabricator.wikimedia.org/T356766 (10jijiki) We'll take a look and get back to you [16:56:13] akosiaris: I mean the function should return the URL to the listener, not to the service - might be wrong [16:56:23] ah, shit. I gtg. Will follow up [16:58:31] jayme: yes, it returns localhost:6052 [16:58:42] which is indeed parsoid-async, but the backing service is parsoid-php [16:59:10] I am wondering a bit what is going on in that graph, but it's getting late and my brain isn't at it's best [17:44:00] * duesen blinks [17:44:11] sorry, I got lost :)