[00:04:01] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:rack/setup/install 3 sessionstore hosts (eqiad) - https://phabricator.wikimedia.org/T349875 (10Eevans) [00:04:55] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:rack/setup/install 3 sessionstore hosts (eqiad) - https://phabricator.wikimedia.org/T349875 (10Eevans) 05Open→03Resolved done. [09:01:24] 10serviceops, 10Kubernetes: Outage of wikikube codfw apiservers - https://phabricator.wikimedia.org/T353233 (10ops-monitoring-bot) VM kubemaster2001.codfw.wmnet rebooted by jayme@cumin2002 with reason: increase vCPUs from 2 to 4 [09:45:58] 10serviceops, 10CX-cxserver, 10RESTBase Sunsetting, 10Language-Team (Language-2023-October-December), 10Patch-For-Review: Make cxserver call parsoid endpoints on MediaWiki, instead of going through RESTbase - https://phabricator.wikimedia.org/T344982 (10MSantos) @santhosh is there anything left for this... [11:24:42] 10serviceops, 10Kubernetes: Outage of wikikube codfw apiservers - https://phabricator.wikimedia.org/T353233 (10ops-monitoring-bot) VM kubemaster2002.codfw.wmnet rebooted by jayme@cumin2002 with reason: increase vCPUs from 2 to 4 [11:26:08] 10serviceops, 10Kubernetes: Outage of wikikube codfw apiservers - https://phabricator.wikimedia.org/T353233 (10ops-monitoring-bot) VM kubemaster1001.eqiad.wmnet rebooted by jayme@cumin2002 with reason: increase vCPUs from 2 to 4 [11:43:18] 10serviceops, 10Kubernetes, 10Patch-For-Review: Outage of wikikube codfw apiservers - https://phabricator.wikimedia.org/T353233 (10ops-monitoring-bot) VM kubemaster1002.eqiad.wmnet rebooted by jayme@cumin2002 with reason: increase vCPUs from 2 to 4 [15:59:06] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Alert on calico components being down - https://phabricator.wikimedia.org/T353463 (10JMeybohm) p:05Triage→03High [16:09:08] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Migtate wikikube control planes to hardware nodes - https://phabricator.wikimedia.org/T353464 (10JMeybohm) p:05Triage→03Medium [16:11:19] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Migrate wikikube control planes to hardware nodes - https://phabricator.wikimedia.org/T353464 (10JMeybohm) [16:18:53] 10serviceops, 10MediaWiki-DjVu, 10Shellbox, 10Structured-Data-Backlog, and 3 others: RuntimeException: firejail is enabled, but cannot be found - https://phabricator.wikimedia.org/T352515 (10brennen) Seeing this also for PdfHandler: ==== Error ==== * service.version: 1.42.0-wmf.9 * trace.id: 9423484c-e7... [16:46:00] 10serviceops, 10MW-on-K8s, 10Discovery-Search (Current work), 10Patch-For-Review: mediawiki k8s jobrunner fails connecting to cloudelastic with a TLS error - https://phabricator.wikimedia.org/T352906 (10akosiaris) Final update, I 've just reverted the "stop the bleeding" patch and now `cirrusSearchCheckerJ... [17:25:12] 10serviceops, 10MW-on-K8s, 10Discovery-Search (Current work): mediawiki k8s jobrunner fails connecting to cloudelastic with a TLS error - https://phabricator.wikimedia.org/T352906 (10akosiaris) 05Open→03Resolved a:03akosiaris https://grafana-rw.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?orgId=1&v... [17:25:17] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Move MediaWiki jobs to mw-on-k8s - https://phabricator.wikimedia.org/T349796 (10akosiaris) [17:36:38] 10serviceops, 10CX-cxserver, 10RESTBase Sunsetting, 10Language-Team (Language-2023-October-December), 10Patch-For-Review: Make cxserver call parsoid endpoints on MediaWiki, instead of going through RESTbase - https://phabricator.wikimedia.org/T344982 (10Nikerabbit) I believe the review and deployment of...