[00:24:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [00:51:14] 10Tool-etherpad-backup: Setup proof of concept storage and retrieval from WMCS object storage - https://phabricator.wikimedia.org/T422958#11810227 (10bd808) I followed the directions from https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide#S3_API to create credentials for the [[https://ldap.toolfo... [00:56:17] 10Tool-etherpad-backup: Setup proof of concept storage and retrieval from WMCS object storage - https://phabricator.wikimedia.org/T422958#11810229 (10bd808) I tried using `s3cmd ws-create s3://html` to make a bucket that would act as a webservice and automatically serve up an index.html file. Unfortunately the c... [04:44:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [04:46:25] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [05:41:25] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [06:10:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [07:14:26] (03CR) 10MusikAnimal: "Frontend changes are still unreviewed, but this enough for one round of feedback, I assume! Most of it mere nitpicks. Wikisource is still " [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) (owner: 10Alien4444) [07:40:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [08:08:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [08:37:50] 10Tool-humaniki-2: Wrap API responses - https://phabricator.wikimedia.org/T422987 (10Danya) 03NEW [08:38:11] 10Tool-humaniki-2: Wrap API responses - https://phabricator.wikimedia.org/T422987#11810445 (10Danya) p:05Triage→03High [09:30:11] 10Tool-humaniki-2: Wrap API responses - https://phabricator.wikimedia.org/T422987#11810493 (10Danya) 05Open→03In progress [09:56:10] 10VPS-project-Codesearch: Codesearch stuck at Feb 12th? - https://phabricator.wikimedia.org/T421147#11810517 (10A_smart_kitten) Is this reoccuring (or e.g. did it never stop occurring, at least for `operations/mediawiki-config` in the 'Everything' group)? https://codesearch.wmcloud.org/search/?q=MetricsPlat... [10:20:33] 06cloud-services-team, 10Toolforge: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error - https://phabricator.wikimedia.org/T422538#11810535 (10Nokib_Sarkar) Hi, any updates? I cannot start the webservice event today for campwiz. is it possible to start by you on behalf of me from backend? [10:21:11] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280372486 (https://github.com/cluebotng/component-configs/commits/5b34645dff3f37bc9f974635e03cd6b8436f37d1) [10:21:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [10:26:09] (03open) 10danyya: Draft: Wrap api responses [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/5 [10:29:55] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280508671 (https://github.com/cluebotng/component-configs/commits/5953d3fb9c5e414df6995740382b7bd3be49ced2) [10:29:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [10:30:23] 10Tool-humaniki-2: Add explicit return types - https://phabricator.wikimedia.org/T422990 (10Danya) 03NEW [10:30:34] 10Tool-humaniki-2: Add explicit return types - https://phabricator.wikimedia.org/T422990#11810549 (10Danya) p:05Triage→03Low [10:30:38] 10Tool-humaniki-2: Add explicit return types - https://phabricator.wikimedia.org/T422990#11810551 (10Danya) 05Open→03In progress [10:38:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [11:18:52] 06cloud-services-team, 10Toolforge: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error - https://phabricator.wikimedia.org/T422538#11810576 (10HakanIST) @Nokib_Sarkar Just a note on the latest traceback: the 409 Conflict on the `/deployments` endpoint usually means the deployment already exists... [11:25:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [12:12:01] 06cloud-services-team, 10Toolforge: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error - https://phabricator.wikimedia.org/T422538#11810618 (10Nokib_Sarkar) @HakanIST ` tools.campwiz-backend@tools-bastion-15:~$ toolforge webservice buildservice start --mount all Traceback (most recent call las... [12:33:08] 06cloud-services-team, 10Toolforge: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error - https://phabricator.wikimedia.org/T422538#11810691 (10HakanIST) @Nokib_Sarkar try `toolforge webservice buildservice restart --mount all` (with the `buildservice` flag). The plain `toolforge webservice resta... [12:40:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [12:41:25] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [14:02:45] 06cloud-services-team, 10Toolforge: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error - https://phabricator.wikimedia.org/T422538#11810828 (10Nokib_Sarkar) So, the issue was I had a job named `campwiz-backend` using push-to-deploy. My tool name was also `campwiz-backend`. So, when I tried to st... [14:03:48] 06cloud-services-team, 10Toolforge: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error - https://phabricator.wikimedia.org/T422538#11810829 (10Nokib_Sarkar) a:03taavi [14:04:04] 06cloud-services-team, 10Toolforge: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error - https://phabricator.wikimedia.org/T422538#11810830 (10Nokib_Sarkar) 05Open→03Resolved [16:05:39] (03update) 10danyya: Draft: Wrap api responses [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/5 [16:11:46] (03update) 10danyya: Draft: Wrap api responses [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/5 [16:13:41] (03update) 10danyya: Wrap api responses [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/5 [16:13:48] (03merge) 10danyya: Wrap api responses [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/5 [16:14:06] 10Tool-humaniki-2: Wrap API responses - https://phabricator.wikimedia.org/T422987#11810930 (10Danya) 05In progress→03Resolved [16:14:07] 10Tool-humaniki-2: Add explicit return types - https://phabricator.wikimedia.org/T422990#11810932 (10Danya) 05In progress→03Resolved [16:16:08] 10Tool-humaniki-2: Retro-active statistics - https://phabricator.wikimedia.org/T422342#11810933 (10Danya) p:05Low→03Triage [16:16:14] 10Tool-humaniki-2: Setup push to deploy - https://phabricator.wikimedia.org/T422347#11810934 (10Danya) p:05Low→03Triage [16:21:37] 10Tool-humaniki-2: Favicon - https://phabricator.wikimedia.org/T423004 (10Danya) 03NEW [16:21:54] 10Tool-humaniki-2: Favicon - https://phabricator.wikimedia.org/T423004#11810962 (10Danya) p:05Triage→03Low [16:35:32] 10Tool-humaniki-2: [Language view] Show stats for all wiki languages - https://phabricator.wikimedia.org/T422334#11810979 (10Danya) 05Open→03In progress [17:10:29] 10Toolforge, 06tools-platform-team: webservice start should give a better error message when a conflicting job exists - https://phabricator.wikimedia.org/T423005 (10taavi) 03NEW [17:10:51] 06cloud-services-team, 10Toolforge: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error - https://phabricator.wikimedia.org/T422538#11811004 (10taavi) 05Resolved→03Invalid a:05taavi→03None Filed T423005 for making the error message better, and marking this as invalid since nothing was... [18:10:04] (03open) 10danyya: Draft: Get stats for all wiki languages [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/6 [19:06:50] 10Tool-wikiloves: wikiloves updates fail with "Unknown column 'cl_to' in 'SELECT'" - https://phabricator.wikimedia.org/T420408#11811146 (10JeanFred) The issue is that `cl_to` was dropped from the wikireplicas mid-March, per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/2L... [19:07:14] 10Tool-wikiloves: wikiloves updates fail with "Unknown column 'cl_to' in 'SELECT'" - https://phabricator.wikimedia.org/T420408#11811148 (10JeanFred) Same as {T421295} [19:26:06] (03PS5) 10Alien4444: Add proofreadpage quality support [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) [19:26:31] (03CR) 10Alien4444: Add proofreadpage quality support (0312 comments) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) (owner: 10Alien4444) [19:36:25] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [19:37:25] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [19:52:39] 10Tool-humaniki-2: Use dumps? - https://phabricator.wikimedia.org/T423012 (10Danya) 03NEW [20:21:34] 10Tool-etherpad-backup: display a list of backed up etherpads somewhere - https://phabricator.wikimedia.org/T423014 (10Novem_Linguae) 03NEW [20:23:39] 10Tool-etherpad-backup: display a list of backed up etherpads somewhere - https://phabricator.wikimedia.org/T423014#11811274 (10Novem_Linguae) [20:48:46] (03update) 10danyya: Draft: Get stats for all wiki languages [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/6 [20:53:54] (03update) 10danyya: Get stats for all wiki languages [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/6 [20:54:00] (03merge) 10danyya: Get stats for all wiki languages [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/6 [20:55:31] 10Tool-humaniki-2: [Language view] Show stats for all wiki languages - https://phabricator.wikimedia.org/T422334#11811285 (10Danya) 05In progress→03Resolved [20:57:24] 10Tool-refill: delete 33,000 unnecessary front end files from production - https://phabricator.wikimedia.org/T422441#11811286 (10Novem_Linguae) 05Open→03Resolved a:03Novem_Linguae ` tools.refill@tools-bastion-15:~$ rm -fr .aptitude tools.refill@tools-bastion-15:~$ rm -fr .cache tools.refill@tools-basti... [20:57:35] 10Tool-refill: delete 33,000 unnecessary front end files from production - https://phabricator.wikimedia.org/T422441#11811289 (10Novem_Linguae) [20:59:29] (03PS1) 10Lokal Profil: switch yaml.load for yaml.safe_load [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1270149 [21:01:52] (03Abandoned) 10Lokal Profil: switch yaml.load for yaml.safe_load [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1270149 (owner: 10Lokal Profil) [21:04:00] (03PS3) 10Lokal Profil: Switch to Python 3.11 as default interpreter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1251441 (https://phabricator.wikimedia.org/T409003) (owner: 10Jean-Frédéric) [21:04:29] (03CR) 10CI reject: [V:04-1] Switch to Python 3.11 as default interpreter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1251441 (https://phabricator.wikimedia.org/T409003) (owner: 10Jean-Frédéric) [21:04:55] (03CR) 10Lokal Profil: "Just a rebase and fixing a pyyaml upgrade related issue" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1251441 (https://phabricator.wikimedia.org/T409003) (owner: 10Jean-Frédéric) [21:28:32] 10Tool-refill: massive error.log file spam due to LabsDB.php PHP fatal error - https://phabricator.wikimedia.org/T389917#11811309 (10Novem_Linguae) I deleted LabsDB.php today as part of {T422441}, so I expect the error.log issue to be solved. I manually deleted uwsgi.log a couple days ago and it hasn't come bac... [21:32:29] (03PS4) 10Lokal Profil: Switch to Python 3.11 as default interpreter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1251441 (https://phabricator.wikimedia.org/T409003) (owner: 10Jean-Frédéric) [21:34:25] (03CR) 10CI reject: [V:04-1] Switch to Python 3.11 as default interpreter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1251441 (https://phabricator.wikimedia.org/T409003) (owner: 10Jean-Frédéric) [21:52:08] 10Tool-humaniki-2: [Birth-years view] Add a form to filter chart by date range - https://phabricator.wikimedia.org/T422339#11811333 (10Danya) p:05Medium→03High [21:52:38] 10Tool-humaniki-2: Setup push to deploy - https://phabricator.wikimedia.org/T422347#11811336 (10Danya) a:05Danya→03None [21:53:00] 10Tool-humaniki-2: Make app title clickable - https://phabricator.wikimedia.org/T422346#11811338 (10Danya) 05Open→03In progress [21:53:01] 10Tool-humaniki-2: Make visualisation menu not selectable - https://phabricator.wikimedia.org/T422345#11811340 (10Danya) 05Open→03In progress [21:53:44] (03PS5) 10Lokal Profil: Switch to Python 3.11 as default interpreter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1251441 (https://phabricator.wikimedia.org/T409003) (owner: 10Jean-Frédéric) [21:53:49] (03open) 10danyya: Draft: UI tweaks [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/7 [22:09:16] (03CR) 10Lokal Profil: [C:03+2] Follow cldr change renaming data files "CldrMain*" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1082452 (https://phabricator.wikimedia.org/T377939) (owner: 10Awight) [22:09:41] (03update) 10danyya: Draft: UI tweaks [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/7 [22:11:11] (03Merged) 10jenkins-bot: Follow cldr change renaming data files "CldrMain*" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1082452 (https://phabricator.wikimedia.org/T377939) (owner: 10Awight) [22:11:58] (03update) 10danyya: UI tweaks [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/7 [22:12:09] (03merge) 10danyya: UI tweaks [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/7 [22:16:24] 10Tool-humaniki-2: Make app title clickable - https://phabricator.wikimedia.org/T422346#11811347 (10Danya) 05In progress→03Resolved [22:16:26] 10Tool-humaniki-2: Make visualisation menu not selectable - https://phabricator.wikimedia.org/T422345#11811349 (10Danya) 05In progress→03Resolved