[00:11:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [00:31:28] 14Grid-Engine-to-K8s-Migration: 14Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - 14https://phabricator.wikimedia.org/T319883#9640650 (10MBH) 14Thank you very much, I will try to rewrite my tools to dotnet app in the coming weeks. But after you updated web server config, people complaining t... [00:32:06] 10wikitech.wikimedia.org: Requesting access for - https://phabricator.wikimedia.org/T360392 (10Rubens522) 03NEW [00:49:21] 10wikitech.wikimedia.org: Requesting access for - https://phabricator.wikimedia.org/T360392#9640703 (10JJMC89) a:05Rubens522→03None [02:20:14] 06Toolforge-standards-committee: 14Adoption request for Muninnbot - 14https://phabricator.wikimedia.org/T358897#9640812 (10Frostly) 05Stalled→03Invalid 14Tigraan has stepped back in and provided access :) [03:11:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [03:55:15] 10Toolforge (Software install/update): No module named - https://phabricator.wikimedia.org/T360398 (10Kaleem_Bhatti) 03NEW [06:11:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [09:10:47] 10Toolforge (Toolforge iteration 07): [harbor] upgrade to 2.10.x - https://phabricator.wikimedia.org/T354507#9640929 (10Slst2020) 05Stalled→03Open [09:10:55] 10Toolforge (Toolforge iteration 07): [builds-cli,builds-api] `build quota` fails if tool has no builds - https://phabricator.wikimedia.org/T353701#9640930 (10Slst2020) [09:11:03] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review, 07Upstream: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417#9640931 (10Slst2020) [09:11:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [09:12:03] 10Toolforge (Toolforge iteration 07): [harbor] upgrade to 2.10.1 - https://phabricator.wikimedia.org/T354507#9640932 (10Slst2020) a:03Slst2020 [09:24:23] 10Cloud-VPS: Request to add catalyst.wmcloud.org webproxy subdomain for the catalyst CloudVPS project - https://phabricator.wikimedia.org/T360364#9641031 (10taavi) a:03taavi [09:28:00] 10Cloud-VPS: 14Request to add catalyst.wmcloud.org webproxy subdomain for the catalyst CloudVPS project - 14https://phabricator.wikimedia.org/T360364#9641073 (10taavi) 05Open→03Resolved [09:28:26] 10wikitech.wikimedia.org, 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review, and 2 others: wikitech hook to disable Gerrit user uses partial matches to identify account - https://phabricator.wikimedia.org/T307558#9641074 (10hashar) With https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/... [09:30:14] 06cloud-services-team, 10wikitech.wikimedia.org: Cleanup logging and curl use in wikitech post-block hooks - https://phabricator.wikimedia.org/T222209#9641083 (10hashar) I think it is related to this task, when amending the hook, I found an issue that `curl_exec` returns `false` on failure but might return the... [09:35:11] 10Toolforge (Toolforge iteration 07): [cicd,infra] Add python 3.11/bookworm support - https://phabricator.wikimedia.org/T360405 (10dcaro) 03NEW [09:38:06] 10Toolforge (Toolforge iteration 07): [cicd,infra] Add python 3.11/bookworm support - https://phabricator.wikimedia.org/T360405#9641171 (10dcaro) p:05Triage→03Medium [09:38:51] (03PS1) 10Majavah: Allow creating wildcard domains [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1012610 (https://phabricator.wikimedia.org/T360363) [09:39:18] (03CR) 10Majavah: [C:03+2] Reformat with black [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1011277 (owner: 10Majavah) [09:39:22] (03CR) 10Majavah: [C:03+2] Sort instances in instance selector [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1011278 (owner: 10Majavah) [09:39:47] (03CR) 10Majavah: [C:03+2] Allow creating wildcard domains [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1012610 (https://phabricator.wikimedia.org/T360363) (owner: 10Majavah) [09:40:08] (03Merged) 10jenkins-bot: Reformat with black [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1011277 (owner: 10Majavah) [09:40:09] (03Merged) 10jenkins-bot: Sort instances in instance selector [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1011278 (owner: 10Majavah) [09:40:24] (03Merged) 10jenkins-bot: Allow creating wildcard domains [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1012610 (https://phabricator.wikimedia.org/T360363) (owner: 10Majavah) [09:54:25] 10wikitech.wikimedia.org, 10Gerrit, 06Release-Engineering-Team, 07SecTeam-Processed, 07Security: 14wikitech hook to disable Gerrit user uses partial matches to identify account - 14https://phabricator.wikimedia.org/T307558#9641206 (10hashar) 05Open→03Resolved 14Thanks again @bd808 ! [10:17:31] 10Toolforge (Toolforge iteration 07): [builds-builder,builds-admission] Remove direct access to tekton from tools and remove the admission controller - https://phabricator.wikimedia.org/T360329#9641292 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_reques... [10:20:37] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [10:20:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:21:16] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [10:21:17] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [builds-builder,builds-admission] Remove direct access to tekton from tools and remove the admission controller - https://phabricator.wikimedia.org/T360329#9641293 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab... [10:21:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:25:25] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [10:25:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:26:06] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [10:26:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:28:51] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [builds-builder,builds-admission] Remove direct access to tekton from tools and remove the admission controller - https://phabricator.wikimedia.org/T360329#9641328 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolfor... [10:35:51] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [builds-builder,builds-admission] Remove direct access to tekton from tools and remove the admission controller - https://phabricator.wikimedia.org/T360329#9641362 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolfor... [10:58:17] 10VPS-project-Codesearch: Can't search for multi-line regex any more - https://phabricator.wikimedia.org/T358786#9641420 (10thiemowmde) Here is an example. When I try to use the `(?s)` modifier, e.g. in https://codesearch.wmcloud.org/core/?q=(?s)\$mSynonyms;&files=\.php$, the UI immediately claims there would be... [11:17:39] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [builds-builder,builds-admission] Remove direct access to tekton from tools and remove the admission controller - https://phabricator.wikimedia.org/T360329#9641491 (10dcaro) [11:17:55] 10Toolforge (Toolforge iteration 07): 14[cicd,infra] Add python 3.11/bookworm support - 14https://phabricator.wikimedia.org/T360405#9641487 (10dcaro) 05Open→03Resolved [11:21:27] 06cloud-services-team, 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: Toolforge: Introduce grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665#9641500 (10taavi) [11:21:47] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-static-14 [11:22:39] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-static-14 [11:24:52] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS (Debian Buster Deprecation), 10Toolforge, 07Epic, 05Goal: Toolforge: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T311897#9641505 (10taavi) [11:24:55] 06cloud-services-team, 10Toolforge (Toolforge iteration 07): 14Upgrade Toolforge static server (tools-static.wmflabs.org) to Debian Bookworm - 14https://phabricator.wikimedia.org/T311913#9641506 (10taavi) [11:24:57] 06cloud-services-team, 10Toolforge (Toolforge iteration 07): 14Upgrade Toolforge static server (tools-static.wmflabs.org) to Debian Bookworm - 14https://phabricator.wikimedia.org/T311913#9641504 (10taavi) 05In progress→03Resolved [11:25:04] 10Toolforge: Upgrade toolsbeta-nfs to Debian Bullseye/Bookworm - https://phabricator.wikimedia.org/T360419 (10taavi) 03NEW [11:25:51] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:29:28] (InstanceDown) firing: Project tools instance tools-static-14 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:30:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:31:20] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:31:56] tools-static-14 is me [11:34:28] (InstanceDown) resolved: Project tools instance tools-static-14 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:36:20] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:39:20] (ProbeDown) firing: (2) Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:44:20] (ProbeDown) resolved: (2) Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:46:20] (ProbeDown) firing: (2) Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:51:20] (ProbeDown) resolved: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:58:01] 14Grid-Engine-to-K8s-Migration: 14Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - 14https://phabricator.wikimedia.org/T320164#9641619 (10dcaro) 14>>! In T320164#9635399, @MBH wrote: > New errors on `autopurge-daily`: > ` > WARNING: API error protectedpage: This page has been protec... [11:59:26] 10Cloud-VPS: 14Support wildcard host matching web proxies in per-project zones - 14https://phabricator.wikimedia.org/T360363#9641620 (10taavi) 05Open→03Resolved 14This should now be possible. Please ping me on IRC if it doesn't work as expected. [12:10:41] (CloudVPSDesignateLeaks) firing: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:11:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [12:13:50] (ProbeDown) firing: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:15:41] (CloudVPSDesignateLeaks) firing: (5) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:18:50] (ProbeDown) resolved: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:24:20] (ProbeDown) firing: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:29:20] (ProbeDown) resolved: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:54:47] 14Toolforge (Toolforge iteration 02), 14Toolforge Build Service: 14[harbor] Investigate new robot account permissions in Harbor 2.10.0 - 14https://phabricator.wikimedia.org/T354270#9641693 (10Slst2020) [13:03:36] (03PS1) 10Josefanthony: Bug: T343438 Fix typo on user setting route [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1012648 (https://phabricator.wikimedia.org/T343438) [13:11:59] (03CR) 10Josefanthony: "I was able to fix the typo error by correcting the comment in the routes.py from # In this case at least on language repeats... to # In th" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1012648 (https://phabricator.wikimedia.org/T343438) (owner: 10Josefanthony) [13:20:41] (CloudVPSDesignateLeaks) firing: (5) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:25:41] (CloudVPSDesignateLeaks) resolved: (5) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:30:18] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.drain_node (T348643) [13:30:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:30:24] T348643: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643 [14:07:15] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS (Debian Buster Deprecation), 10Toolforge, 07Epic, 05Goal: Toolforge: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T311897#9642091 (10taavi) [14:07:18] 06cloud-services-team, 10Toolforge (Toolforge iteration 07): 14Upgrade Toolforge Kubernetes to version 1.24 - 14https://phabricator.wikimedia.org/T307651#9642092 (10taavi) [14:08:08] 06cloud-services-team, 10Toolforge (Toolforge iteration 07): Toolforge: Introduce grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665#9642090 (10taavi) 05Open→03In progress [14:09:32] 10Toolforge (Toolforge iteration 07): 14Support monorepos with the Multi Procfile buildpack - 14https://phabricator.wikimedia.org/T355329#9642105 (10dcaro) 05Open→03Resolved 14Closing this as resolved, please reopen if you still see issues. [14:21:03] 10Toolforge: [jobs-api,buildservice-api,envvars-api] evaluate crossplane for composite objects creation and maintenance - https://phabricator.wikimedia.org/T360016#9642161 (10dcaro) p:05Triage→03Medium [14:58:16] 10Toolforge (Toolforge iteration 07): [harbor] upgrade to 2.10.1 - https://phabricator.wikimedia.org/T354507#9642509 (10dcaro) Yay https://github.com/goharbor/harbor/releases/tag/v2.10.1 [15:11:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [15:21:51] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T348643) [15:21:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:21:57] T348643: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643 [15:25:04] PROBLEM - Host wikitech-static.wikimedia.org is DOWN: PING CRITICAL - Packet loss = 100% [15:49:09] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.drain_node (T348643) [15:49:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:49:15] T348643: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643 [15:52:57] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643#9642847 (10dcaro) [16:25:29] 10wikitech.wikimedia.org: 14Requesting access for  - 14https://phabricator.wikimedia.org/T360392#9642972 (10bd808) 05Open→03Declined 14* No specific rights requested * Developer account is only 1 day old and not linked to any established SUL identity * User obviously doing edits to thei... [16:27:17] RECOVERY - Host wikitech-static.wikimedia.org is UP: PING OK - Packet loss = 0%, RTA = 22.21 ms [18:11:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [18:18:26] 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): Migrate per-project Puppet servers to Puppet 7 - https://phabricator.wikimedia.org/T351452#9643416 (10Andrew) [18:28:15] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T348643) [18:28:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:28:22] T348643: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643 [18:36:24] 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): Update gitlab-runners project puppetmaster - https://phabricator.wikimedia.org/T360459 (10Andrew) 03NEW [18:38:55] 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): Update gitlab-runners project puppetmaster - https://phabricator.wikimedia.org/T360459#9643535 (10Andrew) [18:40:06] 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): Update gitlab-runners project puppetmaster - https://phabricator.wikimedia.org/T360459#9643538 (10Andrew) [18:48:48] (03PS1) 10Catrope: releases: Bump Codex to 1.3.5 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1012739 [18:50:58] 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): Update gitlab-runners project puppetmaster - https://phabricator.wikimedia.org/T360459#9643592 (10dancy) @Jelto @eoghan @Arnoldokoth Looks like your department. [18:51:44] 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): Update Integration project puppetmaster - https://phabricator.wikimedia.org/T360461 (10Andrew) 03NEW [18:55:14] 06cloud-services-team, 10VPS-Projects, 06Release-Engineering-Team, 10Puppet (Puppet 7.0): Update Integration project puppetmaster - https://phabricator.wikimedia.org/T360461#9643634 (10dancy) [19:31:50] (ProbeDown) firing: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:36:50] (ProbeDown) resolved: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:40:34] 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470 (10Andrew) 03NEW [19:41:16] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops: hw troubleshooting: /dev/sdg disk not working properly in cloudcephosd1017.eqiad.wmnet - https://phabricator.wikimedia.org/T359049#9643822 (10Jclark-ctr) @dcaro server is out of warranty i did replace disk with an extra one we had on hand in eqiad please confirm... [19:42:52] (03CR) 10LWatson: [C:03+2] releases: Bump Codex to 1.3.5 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1012739 (owner: 10Catrope) [19:44:02] (03Merged) 10jenkins-bot: releases: Bump Codex to 1.3.5 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1012739 (owner: 10Catrope) [20:17:48] 06cloud-services-team, 10VPS-Projects, 06Release-Engineering-Team, 10Puppet (Puppet 7.0): Update Integration project puppetmaster - https://phabricator.wikimedia.org/T360461#9643977 (10hashar) a:05hashar→03None [20:18:52] 06cloud-services-team, 10VPS-Projects, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Puppet (Puppet 7.0): Update Integration project puppetmaster - https://phabricator.wikimedia.org/T360461#9643984 (10hashar) [20:29:32] 10Toolforge: `webservice` should have more easily understandable error messages when run as a non-tool user - https://phabricator.wikimedia.org/T360478 (10bd808) 03NEW [20:31:22] 06cloud-services-team, 10VPS-Projects, 06collaboration-services, 10Puppet (Puppet 7.0): Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9644022 (10Dzahn) [21:11:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [21:20:47] (03PS1) 10Majavah: Add python3 back to images that used to have it via webservice-runner [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1012753 [21:23:06] (03CR) 10BryanDavis: [C:03+1] "It would be nice to put the python cgi runner block conditionally back in shared/lighttpd/webservice-runner too." [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1012753 (owner: 10Majavah) [21:25:59] (03CR) 10Majavah: [C:03+2] "If you mean" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1012753 (owner: 10Majavah) [21:26:33] (03Merged) 10jenkins-bot: Add python3 back to images that used to have it via webservice-runner [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1012753 (owner: 10Majavah) [22:03:08] 06cloud-services-team, 10VPS-Projects, 06collaboration-services, 10Puppet (Puppet 7.0): Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9644219 (10brennen) a:03brennen Tentatively: I can take a crack at this. [22:28:50] (ProbeDown) firing: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:33:50] (ProbeDown) resolved: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:58:35] 10Toolforge: Missing packages on dev.toolforge.org - https://phabricator.wikimedia.org/T360488 (10Anomie) 03NEW