[00:00:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:07:08] 10Toolforge: [envvars-cli] Add option to not show envvar values when listing - https://phabricator.wikimedia.org/T363544 (10LucasWerkmeister) 03NEW [00:09:11] 10Toolforge: [envvars-cli] Add option to not show envvar values when listing - https://phabricator.wikimedia.org/T363544#9746941 (10LucasWerkmeister) Two use-cases I have in mind: - debugging a tool with someone else who is “shoulder surfing” - publicly documenting the defined envvars (“the tool expects the fol... [00:16:45] (WidespreadPuppetFailure) firing: Puppet has failed on wmcs cluster - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=3&var-cluster=wmcs - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [00:17:48] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on cloudcontrol2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [00:58:46] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:49:24] 10Toolforge: Query appears to run for a longer time when invoked via toolforge jobs framework - https://phabricator.wikimedia.org/T363286#9746999 (10Huji) fawiki and I believe web replicas. Maybe [[https://github.com/PersianWikipedia/fawikibot/blob/master/HujiBot/stats.py#L151|this line]] will help you verify my... [02:07:55] 10Tool-bridgebot, 07Upstream: Bridgebot freaks out and sends double messages from IRC to Telegram - https://phabricator.wikimedia.org/T305487#9747053 (10bd808) `lang=irc [02:04] wm-bb (~wm-bridge@wikimedia/bot/wm-bridgebot) left IRC (Quit: Bouncer quit) [02:04] wm-bb (~wm-bridge@wikimedia/bot/wm-bridgebot) j... [03:01:55] (ToolforgeKubernetesCapacity) firing: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [03:13:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:21:55] (ToolforgeKubernetesCapacity) resolved: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [03:28:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:16:45] (WidespreadPuppetFailure) firing: Puppet has failed on wmcs cluster - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=3&var-cluster=wmcs - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [04:17:48] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on cloudcontrol2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:47:41] (CloudVPSDesignateLeaks) firing: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:52:41] (CloudVPSDesignateLeaks) firing: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:57:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:58:15] (WidespreadPuppetFailure) resolved: Puppet has failed on wmcs cluster - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=3&var-cluster=wmcs - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [05:03:46] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [05:09:18] (PuppetZeroResources) resolved: (2) Puppet has failed generate resources on cloudcontrol2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:22:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:58:27] 06cloud-services-team, 10Horizon, 10Striker: Striker/Horizon are running in Blubber built containers with a runtime UID that does not exist on the host machine - https://phabricator.wikimedia.org/T363452#9747146 (10MoritzMuehlenhoff) >>! In T363452#9744332, @taavi wrote: > Those are running in containers, an... [08:13:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:23:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:42:48] 10Tools: zinbot not patrolling - https://phabricator.wikimedia.org/T363552 (10TheresNoTime) 03NEW [08:55:59] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [08:56:09] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [08:56:50] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [08:57:00] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [08:59:14] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [08:59:22] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [08:59:32] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [08:59:41] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [09:00:09] 10Toolforge (Quota-requests), 13Patch-For-Review: Request increased quota for uploadmap Toolforge tool - https://phabricator.wikimedia.org/T362975#9747192 (10CodeReviewBot) aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/267 maintain-kubeusers: bump service... [09:05:20] 10Toolforge (Quota-requests), 13Patch-For-Review: Request increased quota for uploadmap Toolforge tool - https://phabricator.wikimedia.org/T362975#9747207 (10aborrero) 05Open→03Resolved [09:07:57] 06cloud-services-team, 10Toolforge: toolforge lima-kilo: refresh maintain-kubeusers test data - https://phabricator.wikimedia.org/T363482#9747209 (10aborrero) 05Open→03In progress p:05Triage→03Medium [09:08:46] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [09:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:52:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:18:54] 10Tools: zinbot not patrolling - https://phabricator.wikimedia.org/T363552#9747361 (10TheresNoTime) I made a fork with very verbose logging — seems the normal `.out` wasn't logging stdout? Its //running//, but there's nothing to review? Does that sound right..? ` ==> /data/project/zinbot/zinbot-oneoff.out <== S... [11:10:55] (ToolforgeKubernetesCapacity) firing: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [11:20:58] 06cloud-services-team, 10Toolforge: toolforge lima-kilo: refresh maintain-kubeusers test data - https://phabricator.wikimedia.org/T363482#9747572 (10CodeReviewBot) aborrero updated https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/19 maintain-kubeusers: adapt to lima-kilo [11:29:43] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete dummy certs [labs/private] - 10https://gerrit.wikimedia.org/r/1024421 (https://phabricator.wikimedia.org/T360439) (owner: 10Muehlenhoff) [11:35:41] 10Tool-global-search: Global search can not find some pages in zhwiki - https://phabricator.wikimedia.org/T363568 (10Bugreporter) 03NEW [12:03:46] (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [12:12:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:22:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:18:27] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 05Goal: Upgrade cloud-vps openstack to version 'Bobcat' - https://phabricator.wikimedia.org/T356287#9747900 (10Andrew) 05Open→03Resolved a:03Andrew [14:14:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:24:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:14:18] 10Tool-global-search: Global search can not find redirects by title - https://phabricator.wikimedia.org/T363568#9748188 (10Bugreporter) [15:15:21] 10Tool-global-search: Global search can not find redirects by title - https://phabricator.wikimedia.org/T363568#9748192 (10Bugreporter) Oh, I found the specific cause and thus updated the title. This is still a valid task; see task linked for why I want to search them. [15:17:41] (CloudVPSDesignateLeaks) firing: (3) Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:25:57] (03PS1) 10JHathaway: postfix: mx-{in,out} test data [labs/private] - 10https://gerrit.wikimedia.org/r/1024726 [15:27:24] (03PS2) 10JHathaway: postfix: mx-{in,out} test data [labs/private] - 10https://gerrit.wikimedia.org/r/1024726 (https://phabricator.wikimedia.org/T325398) [15:36:44] (03CR) 10JHathaway: [C:03+2] postfix: mx-{in,out} test data [labs/private] - 10https://gerrit.wikimedia.org/r/1024726 (https://phabricator.wikimedia.org/T325398) (owner: 10JHathaway) [15:36:50] (03CR) 10JHathaway: [V:03+2 C:03+2] postfix: mx-{in,out} test data [labs/private] - 10https://gerrit.wikimedia.org/r/1024726 (https://phabricator.wikimedia.org/T325398) (owner: 10JHathaway) [15:52:46] 10Openstack-Magnum: Deploy k8s greater than 1.23 - https://phabricator.wikimedia.org/T363504#9748349 (10rook) labels = { #kube_tag = "v1.26.15-rancher1-linux-amd64" kube_tag = "v1.26.8-rancher1" #flannel_tag = "v0.21.5" cinder_csi_enabled = "true" container_runtime = "containe... [16:22:41] (CloudVPSDesignateLeaks) firing: (3) Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:22:56] (CloudVPSDesignateLeaks) firing: (3) Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:27:00] (03PS1) 10Andrea Denisse: ssl: Delete dummy TLS key for the Prometheus hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1024712 (https://phabricator.wikimedia.org/T360414) [16:27:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:30:56] (03CR) 10Andrea Denisse: [C:03+2] ssl: Delete dummy TLS key for the Prometheus hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1024712 (https://phabricator.wikimedia.org/T360414) (owner: 10Andrea Denisse) [16:30:58] (03CR) 10Andrea Denisse: [V:03+2 C:03+2] ssl: Delete dummy TLS key for the Prometheus hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1024712 (https://phabricator.wikimedia.org/T360414) (owner: 10Andrea Denisse) [16:35:32] (03PS1) 10Andrew Bogott: Update ugettext_lazy and ungettext_lazy for modern django [openstack/horizon/wmf-sudo-dashboard] - 10https://gerrit.wikimedia.org/r/1024745 [16:44:25] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Update ugettext_lazy and ungettext_lazy for modern django [openstack/horizon/wmf-sudo-dashboard] - 10https://gerrit.wikimedia.org/r/1024745 (owner: 10Andrew Bogott) [16:48:01] (03PS1) 10Andrew Bogott: Update ugettext_lazy and ungettext_lazy for modern django [openstack/horizon/wmf-member-dashboard] - 10https://gerrit.wikimedia.org/r/1024766 [16:50:56] (03PS1) 10Andrew Bogott: Update ugettext_lazy and ungettext_lazy for modern django [openstack/horizon/wmf-member-dashboard] - 10https://gerrit.wikimedia.org/r/1024768 [16:51:13] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Update ugettext_lazy and ungettext_lazy for modern django [openstack/horizon/wmf-member-dashboard] - 10https://gerrit.wikimedia.org/r/1024768 (owner: 10Andrew Bogott) [16:51:24] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Update ugettext_lazy and ungettext_lazy for modern django [openstack/horizon/wmf-member-dashboard] - 10https://gerrit.wikimedia.org/r/1024766 (owner: 10Andrew Bogott) [16:54:19] (03PS1) 10Andrew Bogott: Update ugettext_lazy and ungettext_lazy for modern django [openstack/horizon/wmf-puppet-dashboard] - 10https://gerrit.wikimedia.org/r/1024770 [16:54:33] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Update ugettext_lazy and ungettext_lazy for modern django [openstack/horizon/wmf-puppet-dashboard] - 10https://gerrit.wikimedia.org/r/1024770 (owner: 10Andrew Bogott) [16:56:36] (03PS1) 10Andrew Bogott: Update ugettext_lazy and ungettext_lazy for modern django [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1024771 [16:57:42] (03PS1) 10Andrew Bogott: One more ugettext/gettxt replacement [openstack/horizon/wmf-member-dashboard] - 10https://gerrit.wikimedia.org/r/1024772 [16:58:06] (03PS1) 10Andrew Bogott: One more ugettext/gettxt replacement [openstack/horizon/wmf-member-dashboard] - 10https://gerrit.wikimedia.org/r/1024774 [16:58:22] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] One more ugettext/gettxt replacement [openstack/horizon/wmf-member-dashboard] - 10https://gerrit.wikimedia.org/r/1024774 (owner: 10Andrew Bogott) [16:58:33] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] One more ugettext/gettxt replacement [openstack/horizon/wmf-member-dashboard] - 10https://gerrit.wikimedia.org/r/1024772 (owner: 10Andrew Bogott) [16:58:44] (03CR) 10Andrew Bogott: [C:03+2] Update ugettext_lazy and ungettext_lazy for modern django [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1024771 (owner: 10Andrew Bogott) [17:07:00] (03PS1) 10Andrew Bogott: Replace ugettext_lazy imports with gettext_lazy [openstack/horizon/barbican-ui] - 10https://gerrit.wikimedia.org/r/1024777 [17:07:24] (03PS1) 10Andrew Bogott: Replace ugettext_lazy imports with gettext_lazy [openstack/horizon/barbican-ui] - 10https://gerrit.wikimedia.org/r/1024779 [17:07:46] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Replace ugettext_lazy imports with gettext_lazy [openstack/horizon/barbican-ui] - 10https://gerrit.wikimedia.org/r/1024777 (owner: 10Andrew Bogott) [17:07:55] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Replace ugettext_lazy imports with gettext_lazy [openstack/horizon/barbican-ui] - 10https://gerrit.wikimedia.org/r/1024779 (owner: 10Andrew Bogott) [17:25:58] 10PAWS: jupyterlab to 4.1.7 - https://phabricator.wikimedia.org/T363596 (10rook) 03NEW [17:29:50] 10PAWS: jupyterlab to 4.1.7 - https://phabricator.wikimedia.org/T363596#9748878 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/402 [17:30:00] vivian-rook opened https://github.com/toolforge/paws/pull/402 [17:34:15] (03PS1) 10Andrew Bogott: Fix .gitreview [openstack/horizon/trove-dashboard] (2024.1) - 10https://gerrit.wikimedia.org/r/1024784 [18:13:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:23:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:51:05] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Fix .gitreview [openstack/horizon/trove-dashboard] (2024.1) - 10https://gerrit.wikimedia.org/r/1024784 (owner: 10Andrew Bogott) [18:57:13] 10Toolforge: Slow loading on Toolforge - https://phabricator.wikimedia.org/T362822#9749512 (10GPSLeo) This seems to be a problem of the Deutsche Telekom network. I asked them for a statement on this. I think this can therefore become closed. But if this is not fixed soon it might be good to document this problem... [19:18:19] (03CR) 10Jforrester: "The new breaking change went out in 0.0.13; I think we need to pin to 0.0.12 for now. Note that Jon is working on a skip feature so we can" [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1024487 (owner: 10VolkerE) [19:30:55] (ToolforgeKubernetesCapacity) resolved: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [19:32:25] (ToolforgeKubernetesCapacity) firing: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [19:38:40] (03PS1) 10Andrew Bogott: Replace django.conf.urls.url with dhango.urls.re_path [openstack/horizon/wmf-member-dashboard] - 10https://gerrit.wikimedia.org/r/1024794 [19:40:18] (03PS2) 10Andrew Bogott: Replace django.conf.urls.url with django.urls.re_path [openstack/horizon/wmf-member-dashboard] - 10https://gerrit.wikimedia.org/r/1024794 [19:40:28] (03PS1) 10Andrew Bogott: Replace django.conf.urls.url with django.urls.re_path [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1024795 [19:42:25] (ToolforgeKubernetesCapacity) resolved: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [19:42:32] (03PS1) 10Andrew Bogott: Replace django.conf.urls.url with django.urls.re_path [openstack/horizon/wmf-puppet-dashboard] - 10https://gerrit.wikimedia.org/r/1024796 [19:43:42] (03PS1) 10Andrew Bogott: Replace django.conf.urls.url with django.urls.re_path [openstack/horizon/wmf-sudo-dashboard] - 10https://gerrit.wikimedia.org/r/1024797 [19:44:28] (03CR) 10Andrew Bogott: [C:03+2] Replace django.conf.urls.url with django.urls.re_path [openstack/horizon/wmf-puppet-dashboard] - 10https://gerrit.wikimedia.org/r/1024796 (owner: 10Andrew Bogott) [19:44:38] (03CR) 10Andrew Bogott: [C:03+2] Replace django.conf.urls.url with django.urls.re_path [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1024795 (owner: 10Andrew Bogott) [19:44:49] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Replace django.conf.urls.url with django.urls.re_path [openstack/horizon/wmf-member-dashboard] - 10https://gerrit.wikimedia.org/r/1024794 (owner: 10Andrew Bogott) [19:45:00] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Replace django.conf.urls.url with django.urls.re_path [openstack/horizon/wmf-sudo-dashboard] - 10https://gerrit.wikimedia.org/r/1024797 (owner: 10Andrew Bogott) [19:47:11] (03PS1) 10Andrew Bogott: Replace django.conf.urls.url with django.urls.re_path [openstack/horizon/barbican-ui] - 10https://gerrit.wikimedia.org/r/1024799 [19:48:17] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Replace django.conf.urls.url with django.urls.re_path [openstack/horizon/barbican-ui] - 10https://gerrit.wikimedia.org/r/1024799 (owner: 10Andrew Bogott) [20:31:55] (ToolforgeKubernetesCapacity) firing: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [20:41:59] (03PS1) 10Eevans: cassandra: add (faux) password for cassandra-devel user [labs/private] - 10https://gerrit.wikimedia.org/r/1024805 (https://phabricator.wikimedia.org/T355730) [20:44:41] (03CR) 10Eevans: [V:03+2 C:03+2] cassandra: add (faux) password for cassandra-devel user [labs/private] - 10https://gerrit.wikimedia.org/r/1024805 (https://phabricator.wikimedia.org/T355730) (owner: 10Eevans) [20:56:55] (ToolforgeKubernetesCapacity) resolved: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [21:41:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:46:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:56:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:29:55] (ToolforgeKubernetesCapacity) firing: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [22:44:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:54:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:54:55] (ToolforgeKubernetesCapacity) resolved: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity