[03:50:29] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-harbor-2 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:34:18] FIRING: KernelErrors: Server cloudlb1001 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudlb1001 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [09:34:24] 06cloud-services-team: KernelErrors Server cloudlb1001 logged kernel errors - https://phabricator.wikimedia.org/T401543 (10phaultfinder) 03NEW [13:34:32] FIRING: KernelErrors: Server cloudlb1001 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudlb1001 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [14:08:05] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [14:13:04] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-39 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [14:39:18] FIRING: [4x] KernelErrors: Server cloudgw1003 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [14:39:28] 06cloud-services-team: KernelErrors - https://phabricator.wikimedia.org/T401549 (10phaultfinder) 03NEW [15:12:48] 06cloud-services-team, 10Toolforge: `toolforge jobs logs` has inconsistent ordering - https://phabricator.wikimedia.org/T401552 (10DamianZaremba) 03NEW [17:54:34] 06cloud-services-team, 10Toolforge: `toolforge jobs logs` is limited to 5000 lines - https://phabricator.wikimedia.org/T401553 (10DamianZaremba) 03NEW [18:39:33] FIRING: [4x] KernelErrors: Server cloudgw1003 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [18:53:05] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-36 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [19:08:43] 06cloud-services-team, 10Toolforge: wikihistory.toolforge.org is down - https://phabricator.wikimedia.org/T401558 (10Xqt) 03NEW [19:29:27] 10Tools: wikihistory.toolforge.org is down - https://phabricator.wikimedia.org/T401558#11072598 (10JJMC89) [19:35:38] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Building container images/My first Buildpack Python tool was modified, changed by So9q link https://wikitech.wikimedia.org/w/index.php?diff=2331328 edit summary: /* How to create a basic Flask WSGI webservice */ add information about poetry and latest buildpack [19:36:25] Change on 12wikitech.wikimedia.org a page Help:Toolforge/Building container images/My first Buildpack Python tool was modified, changed by So9q link https://wikitech.wikimedia.org/w/index.php?diff=2331329 edit summary: /* How to create a basic Flask WSGI webservice */ remove old note which is not true anymore [20:08:05] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-36 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [22:39:33] FIRING: [4x] KernelErrors: Server cloudgw1003 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [23:12:14] (03open) 10eliza189: added missing logic to session_post [toolforge-repos/miss-search] (update-cycle-toolforge-testing) - 10https://gitlab.wikimedia.org/toolforge-repos/miss-search/-/merge_requests/12