[02:59:24] (03open) 10tmoney: Strip trailing title qualifiers from search phrases [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/6 [02:59:24] (03update) 10tmoney: Strip trailing title qualifiers from search phrases [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/6 [03:00:45] (03merge) 10tmoney: Strip trailing title qualifiers from search phrases [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/6 [03:10:19] 10Tool-findlinkfast: findlinkfast: Allowlist prose-bearing formatting templates for link insertion - https://phabricator.wikimedia.org/T439343 (10Audiodude) 03NEW [06:54:15] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T437407) [06:54:18] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T437407) [06:54:22] T437407: Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407 [07:39:33] 10PAWS, 10Toolforge, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Dumps NFS mount down - https://phabricator.wikimedia.org/T438062#12367662 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi We have alerting in place for automount, I'm optimisticall... [08:25:55] 10Cloud-VPS, 06tools-infrastructure-team: Error when creating snapshot for instance - https://phabricator.wikimedia.org/T439363 (10Sebastian_Berlin-WMSE) 03NEW [09:05:33] (03CR) 10Btullis: [V:03+2 C:03+2] ceph: Move the dummy STS key to the correct hiera file [labs/private] - 10https://gerrit.wikimedia.org/r/1345228 (https://phabricator.wikimedia.org/T435590) (owner: 10Andrea Denisse) [09:10:50] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [09:11:03] FIRING: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:40:14] 10Toolforge, 06tools-platform-team, 07OKR-Work: Create template for maintainer alerts emails - https://phabricator.wikimedia.org/T437081#12368463 (10fnegri) Added the line above and a few other tweaks. This is a preview of both "firing" and "resolved" alerts for the 3 types of alerts we are supporting at the... [09:53:49] 10Cloud-VPS, 06tools-infrastructure-team: Export puppet role as metric for cloudvps too - https://phabricator.wikimedia.org/T439383 (10fgiunchedi) 03NEW [10:59:34] FIRING: ToolsNfsAlmostFull: Toolforge NFS is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull [11:01:26] (03PS1) 10Volans: cumin: Add dummy insetup role report config [labs/private] - 10https://gerrit.wikimedia.org/r/1345507 [11:06:05] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12368761 (10fgiunchedi) a:05Jclark-ctr→03fgiunchedi Ok thank you @Jclark-ctr ! I will try poking 1055 and see what happens. In th... [11:09:22] 10Tool-inteGraality: Upgrade to Bootstrap 5 - https://phabricator.wikimedia.org/T439032#12368790 (10JeanFred) 05Open→03Resolved [11:11:18] 10Tool-inteGraality: Redesign home page for better discoverability - https://phabricator.wikimedia.org/T439394 (10JeanFred) 03NEW [11:35:29] (03CR) 10Volans: "Related to I0c3f97d7f44097f6e503529c8d72472ae1790282" [labs/private] - 10https://gerrit.wikimedia.org/r/1345507 (owner: 10Volans) [12:41:27] (03open) 10isaranto: liftwing-status: Add status page for public Lift Wing services [toolforge-repos/liftwing-status] - 10https://gitlab.wikimedia.org/toolforge-repos/liftwing-status/-/merge_requests/1 [12:44:48] !log fnegri@cloudcumin1001 paws START - Cookbook wmcs.openstack.cloudvirt.vm_console [12:45:47] !log fnegri@cloudcumin1001 paws END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [12:45:50] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [12:46:03] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [12:46:27] (03update) 10isaranto: liftwing-status: Add status page for public Lift Wing services [toolforge-repos/liftwing-status] - 10https://gitlab.wikimedia.org/toolforge-repos/liftwing-status/-/merge_requests/1 [12:46:58] (03merge) 10isaranto: liftwing-status: Add status page for public Lift Wing services [toolforge-repos/liftwing-status] - 10https://gitlab.wikimedia.org/toolforge-repos/liftwing-status/-/merge_requests/1 [12:54:08] (03CR) 10Elukey: [C:03+1] cumin: Add dummy insetup role report config [labs/private] - 10https://gerrit.wikimedia.org/r/1345507 (owner: 10Volans) [13:05:07] (03open) 10isaranto: liftwing-status: Ignore a single slow check with no earlier run [toolforge-repos/liftwing-status] - 10https://gitlab.wikimedia.org/toolforge-repos/liftwing-status/-/merge_requests/2 [13:05:34] (03merge) 10isaranto: liftwing-status: Ignore a single slow check with no earlier run [toolforge-repos/liftwing-status] - 10https://gitlab.wikimedia.org/toolforge-repos/liftwing-status/-/merge_requests/2 [13:13:56] (03open) 10fnegri: maintainer-alerts: remove tool- prefix [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/72 (https://phabricator.wikimedia.org/T437081) [13:25:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [13:27:04] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Set up confd on Toolforge and project-proxy proxy hosts - https://phabricator.wikimedia.org/T438971#12369422 (10taavi) [13:27:05] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Enable use of requestctl for filtering Toolforge and Cloud VPS web traffic - https://phabricator.wikimedia.org/T409759#12369423 (10taavi) [13:27:08] 06cloud-services-team, 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Block web crawlers from accessing Cloud Services - https://phabricator.wikimedia.org/T226688#12369424 (10taavi) [13:33:39] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12369444 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1004 for host cloudcephosd1055.eqiad.wmnet with OS bookworm [13:36:05] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Deploy hiddenparma application - https://phabricator.wikimedia.org/T439418 (10taavi) 03NEW [13:40:07] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Come up with a backup mechanism for P:wmcs::etcd clusters - https://phabricator.wikimedia.org/T438731#12369476 (10taavi) 05Open→03Resolved [13:40:10] 10Toolforge, 06tools-infrastructure-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 13Patch-For-Review: [etcd,infra] Find a backup solution for the etcd database - https://phabricator.wikimedia.org/T339934#12369479 (10taavi) 05Open→03Resolved [14:18:07] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [14:18:18] 10Cloud-VPS, 06tools-infrastructure-team: Error when creating snapshot for instance - https://phabricator.wikimedia.org/T439363#12369669 (10Andrew) a:03Andrew [14:25:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [14:43:54] 10VPS-project-Codesearch, 06Collaboration-Services, 13Patch-For-Review: Codesearch can't build: ERROR: golang1.15 not found - https://phabricator.wikimedia.org/T438777#12369767 (10LSobanski) [14:45:57] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [14:46:04] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [15:03:56] 06tools-infrastructure-team: Upgrade cloudcumin hosts to Trixie - https://phabricator.wikimedia.org/T436667#12370084 (10Volans) p:05Triage→03Medium a:03Volans I'll take care of this [15:06:41] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12370089 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1004 for host cloudcephosd1055.eqiad.wmnet with OS bookworm ex... [15:56:09] 10VPS-project-Codesearch, 06Collaboration-Services, 13Patch-For-Review: Codesearch can't build: ERROR: golang1.15 not found - https://phabricator.wikimedia.org/T438777#12370320 (10LSobanski) p:05Triage→03Medium [15:59:24] (03update) 10fnegri: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) [15:59:24] (03update) 10fnegri: Do not re-notify old alerts after a restart [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/6 (https://phabricator.wikimedia.org/T432865) [15:59:26] (03update) 10fnegri: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) [15:59:29] (03open) 10fnegri: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) [16:12:46] 10Tool-inteGraality: Dark mode - https://phabricator.wikimedia.org/T439446 (10JeanFred) 03NEW [16:26:32] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12370456 (10mikez-WMF) On rollout - not yet, but we can circle back on this - right now we're trying to focus on wrapping up some wishlist tool updates. I am a... [16:59:13] (03CR) 10Volans: [V:03+2 C:03+2] cumin: Add dummy insetup role report config [labs/private] - 10https://gerrit.wikimedia.org/r/1345507 (owner: 10Volans) [17:06:48] (03Abandoned) 10Andrew Bogott: etcd depool_and_remove: add --shutdown option [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1017884 (owner: 10Andrew Bogott) [17:20:18] 06Toolforge-standards-committee, 06LPL Onboarding and Development, 10Starter Kit, 07Privacy: starterkit.toolforge.org loads third-party resources - https://phabricator.wikimedia.org/T438458#12370685 (10MaryMunyoki) [17:51:34] (03CR) 10Andrew Bogott: [C:03+1] views: Use singular backend property [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1308167 (https://phabricator.wikimedia.org/T429960) (owner: 10Majavah) [17:55:33] 10Tool-Pageviews: NaN error on the Page Views page - https://phabricator.wikimedia.org/T439452 (10Sidevar) 03NEW [19:50:13] (03PS1) 10Andrew Bogott: CreateSnapshot: Fix policy check [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1345644 (https://phabricator.wikimedia.org/T439363) [19:53:56] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Error when creating snapshot for instance - https://phabricator.wikimedia.org/T439363#12371321 (10Andrew) This is 'fixed' by the attached patch, which is to say: policy doesn't allow snapshots, so the bug is Horizon showing the option in the menu.... [20:04:34] RESOLVED: ToolsNfsAlmostFull: Toolforge NFS is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull [21:04:34] (03update) 10fnegri: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) [21:04:35] (03update) 10fnegri: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) [21:21:54] (03open) 10fnegri: alerts-api: Use the root URL of each prometheus host [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1434 (https://phabricator.wikimedia.org/T432865) [21:25:31] (03update) 10fnegri: Notify when an alert fires or is resolved [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/5 (https://phabricator.wikimedia.org/T432865) [21:25:32] (03update) 10fnegri: Do not re-notify old alerts after a restart [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/6 (https://phabricator.wikimedia.org/T432865) [21:25:33] (03update) 10fnegri: mailer: Compose the alert emails from templates [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T437081) [21:25:33] (03update) 10fnegri: Send the notification emails [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 (https://phabricator.wikimedia.org/T432865) [21:25:37] (03open) 10fnegri: Send the notification emails [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 (https://phabricator.wikimedia.org/T432865) [21:46:00] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: How long should archived toolforge tools last? - https://phabricator.wikimedia.org/T439486 (10Andrew) 03NEW [21:46:07] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: How long should archived toolforge tools last? - https://phabricator.wikimedia.org/T439486#12371941 (10Andrew) p:05Triage→03Medium [22:14:32] (03PS1) 10MusikAnimal: 3.26.2 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1345659 [22:15:26] (03CR) 10MusikAnimal: [C:03+2] 3.26.2 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1345659 (owner: 10MusikAnimal) [22:16:15] (03Merged) 10jenkins-bot: 3.26.2 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1345659 (owner: 10MusikAnimal) [23:37:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [23:38:03] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [23:42:59] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [23:43:03] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [23:50:51] (03open) 10amire80: Use consistent ellipsis in messages [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/26