[01:02:52] <icinga-wm>	 PROBLEM - toolschecker: NFS read/writeable on labs instances on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/nfs/home - 324 bytes in 60.008 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker
[01:04:32] <icinga-wm>	 RECOVERY - toolschecker: NFS read/writeable on labs instances on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 37.820 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker
[09:28:58] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Toolforge: improve local kubernetes development setup - https://phabricator.wikimedia.org/T326789#10665779 (10aborrero) 05Open→03Resolved a:03aborrero I think we can consider this task done, as lima-kilo has been de-facto adopted for Toolforge...
[11:12:11] <wikibugs>	 10Toolforge (Quota-requests): Increase RAM quota for mbh tool - https://phabricator.wikimedia.org/T389733 (10MBH) 03NEW
[17:55:03] <wmcs-alerts>	 FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[18:05:03] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[18:34:56] <wikibugs>	 06cloud-services-team, 10Data-Services, 10Projects-Cleanup, 13Patch-For-Review: Archive the operations/debs/bdsync repository - https://phabricator.wikimedia.org/T377882#10666091 (10Jdforrester-WMF)