[00:31:56] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of cpu - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [01:51:56] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of cpu - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [07:31:58] 06cloud-services-team, 10Cloud-VPS, 07Epic, 07IPv6: Support IPv6-only VMs - https://phabricator.wikimedia.org/T404466#11178310 (10taavi) [07:31:59] 06cloud-services-team, 06Infrastructure-Foundations, 07IPv6, 07LDAP: Make ldap-ro service available over IPv6 - https://phabricator.wikimedia.org/T397149#11178311 (10taavi) [07:45:45] (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505) [07:48:32] 06cloud-services-team, 10Toolforge: jobs-api is reporting one-off jobs as continuous - https://phabricator.wikimedia.org/T404490 (10taavi) 03NEW [07:56:31] (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505) [08:27:14] (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505) [08:38:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-1 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [10:11:33] (03open) 10swayamagrahari: fix:adds header in request to avoid 403 error [toolforge-repos/isa] - 10https://gitlab.wikimedia.org/toolforge-repos/isa/-/merge_requests/14 [10:11:55] (03open) 10samwilson: Fixes from upgrade to 1.44 [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/3 [10:12:17] (03update) 10samwilson: Fixes from upgrade to 1.44 [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/3 [10:12:30] (03update) 10samwilson: Fixes from upgrade to 1.44 [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/3 [10:29:43] (03open) 10swayamagrahari: Fix for Wiki loves campaign error [toolforge-repos/isa] - 10https://gitlab.wikimedia.org/toolforge-repos/isa/-/merge_requests/15 [10:47:30] (03update) 10swayamagrahari: fix:adds header in request to avoid 403 error [toolforge-repos/isa] - 10https://gitlab.wikimedia.org/toolforge-repos/isa/-/merge_requests/14 [10:50:17] (03update) 10swayamagrahari: Fix for Wiki loves campaign error [toolforge-repos/isa] - 10https://gitlab.wikimedia.org/toolforge-repos/isa/-/merge_requests/15 [11:50:14] (03merge) 10samwilson: Fixes from upgrade to 1.44 [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/3 [12:03:28] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-1 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [13:03:10] 10VPS-project-voterlists, 06Trust-and-Safety, 03Elections: Create voter list for Wikimedia France board of administrators election - https://phabricator.wikimedia.org/T403809#11178435 (10SD0001) a:03SD0001 I ran the first part of the script (populate-edit-count), which finds the edit counts in the interval... [13:40:23] (03close) 10swayamagrahari: fix:adds header in request to avoid 403 error [toolforge-repos/isa] - 10https://gitlab.wikimedia.org/toolforge-repos/isa/-/merge_requests/14 [13:40:43] (03close) 10swayamagrahari: Fix for Wiki loves campaign error [toolforge-repos/isa] - 10https://gitlab.wikimedia.org/toolforge-repos/isa/-/merge_requests/15 [13:44:55] 10VPS-project-voterlists: Move global-election-list-builder to Gerrit or a shared Gitlab namespace - https://phabricator.wikimedia.org/T404495 (10SD0001) 03NEW [13:58:55] (03open) 10swayamagrahari: fix:adds headers for wikidata api requests [toolforge-repos/isa] - 10https://gitlab.wikimedia.org/toolforge-repos/isa/-/merge_requests/16 [14:05:16] (03open) 10swayamagrahari: fix:wiki loves campaign updated regex [toolforge-repos/isa] - 10https://gitlab.wikimedia.org/toolforge-repos/isa/-/merge_requests/17 [14:21:42] 10VPS-project-voterlists, 06Trust-and-Safety, 03Elections: Create voter list for Wikimedia France board of administrators election - https://phabricator.wikimedia.org/T403809#11178454 (10Lepticed7) Hi. As there is only one candidate, we do not need the vote anymore. So we can close this task. Thanks, and sor... [15:18:07] 10VPS-project-voterlists, 06Trust-and-Safety, 03Elections: Create voter list for Wikimedia France board of administrators election - https://phabricator.wikimedia.org/T403809#11178492 (10SD0001) 05Open→03Invalid [17:56:58] 06cloud-services-team, 10Toolforge: jobs-api is reporting one-off jobs as continuous - https://phabricator.wikimedia.org/T404490#11178566 (10dcaro) I think it's the cli, from a local run with `--debug`: [18:01:53] 06cloud-services-team, 10Toolforge: jobs-cli is reporting one-off jobs as continuous - https://phabricator.wikimedia.org/T404490#11178567 (10taavi) [18:02:07] 06cloud-services-team, 10Toolforge: jobs-cli is reporting one-off jobs as continuous - https://phabricator.wikimedia.org/T404490#11178568 (10taavi) a:03taavi [18:02:56] (03open) 10taavi: list: Fix type of one-off jobs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/127 (https://phabricator.wikimedia.org/T404490) [18:02:59] (03update) 10taavi: list: Fix type of one-off jobs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/127 (https://phabricator.wikimedia.org/T404490) [18:06:38] (03open) 10swayamagrahari: fix:language dropdown now works [toolforge-repos/isa] - 10https://gitlab.wikimedia.org/toolforge-repos/isa/-/merge_requests/18 [19:37:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-43 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [19:47:28] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-proxy-7 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [20:12:28] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-proxy-7 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [20:37:36] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Trust and Safety Product Team: Create voter lists on VPS instead of production - https://phabricator.wikimedia.org/T404506 (10SD0001) 03NEW [20:39:05] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Trust and Safety Product Team: Create voter lists on VPS instead of production - https://phabricator.wikimedia.org/T404506#11178712 (10SD0001) For T403809, I wrapped up the script and ran the first part of it (populate-edit-count). Overall, it wa... [20:42:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-43 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [20:42:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-43 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [23:12:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-43 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [23:17:18] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-43 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [23:26:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-43 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses