[00:10:03] (TfInfraTestApplyFailed) resolved: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:15:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:20:03] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:26:42] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:28:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [01:33:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [04:35:03] (InstanceDown) firing: Project tools instance tools-harbor-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:45:35] (HarborDown) firing: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [04:50:03] (InstanceDown) resolved: Project tools instance tools-harbor-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:50:35] (HarborDown) resolved: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [05:22:01] 10VPS-project-Codesearch, 10Special:NewLexeme revival, 10wmde-wikidata-tech: Please add wmde/new-lexeme-special-page to codesearch index - https://phabricator.wikimedia.org/T351938 (10Ladsgroup) sigh, something broke it: ` Dez 28 05:20:39 codesearch8 write_config.py[11616]: Traceback (most recent call last):... [05:26:42] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [05:29:29] (03PS1) 10Ladsgroup: Skip gitlab.wikibase.nl in parsing of gitmodules [labs/codesearch] - 10https://gerrit.wikimedia.org/r/986203 [05:34:34] (03CR) 10Ladsgroup: [C: 03+2] "We should revisit this later." [labs/codesearch] - 10https://gerrit.wikimedia.org/r/986203 (owner: 10Ladsgroup) [05:35:24] (03Merged) 10jenkins-bot: Skip gitlab.wikibase.nl in parsing of gitmodules [labs/codesearch] - 10https://gerrit.wikimedia.org/r/986203 (owner: 10Ladsgroup) [05:50:49] 10VPS-project-Codesearch, 10Special:NewLexeme revival, 10wmde-wikidata-tech: Please add wmde/new-lexeme-special-page to codesearch index - https://phabricator.wikimedia.org/T351938 (10Ladsgroup) Fixed now: https://codesearch.wmcloud.org/deployed/?q=New+Lexeme+Special+Page+form [06:23:19] (HAProxyBackendUnavailable) firing: HAProxy service nova-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [06:53:57] 10Toolforge-standards-committee: Adoption request for dibot - https://phabricator.wikimedia.org/T354072 (10MBH) [09:26:42] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [09:51:49] 10Grid-Engine-to-K8s-Migration: Migrate checkdictation-fa from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319623 (10Ladsgroup) Is there a way to have hunspell in toolforge? possibly a dedicated image? ` (venv) tools.checkdictation-fa@shell-1703755999:~$ pip install cyhunspe... [10:06:07] 10Grid-Engine-to-K8s-Migration: Migrate checkdictation-fa from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319623 (10Ladsgroup) Okay, migration is done: - Migrated to python 3.11 from 2.7(!) - Removed dependency on pywikibot (it relied on pywikibot compat!!) - Migrated fr... [10:23:20] (HAProxyBackendUnavailable) firing: HAProxy service nova-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [11:02:33] 10Grid-Engine-to-K8s-Migration: Migrate cdrbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319620 (10Muhammed4IT) 05In progress→03Resolved Thanks for your helps, the migration was successfully done. [12:01:31] 10Toolforge-standards-committee: Adoption request for dibot - https://phabricator.wikimedia.org/T354072 (10MBH) @komla Dmitry89 agreed to give me rights to his tool, see https://ru.wikipedia.org/w/index.php?title=User_talk:Dmitry89&curid=1816879&diff=135181623&oldid=135175969 [12:47:17] 10Toolforge-standards-committee: Adoption request for dibot - https://phabricator.wikimedia.org/T354072 (10MBH) 05Open→03Resolved a:03MBH He gave me rights to maintain tool. [13:26:32] 10Grid-Engine-to-K8s-Migration: Migrate checkdictation-fa from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319623 (10Huji) You are the best! [13:26:42] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [14:23:20] (HAProxyBackendUnavailable) firing: HAProxy service nova-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [14:26:27] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [14:37:40] 10Toolforge Build Service: New upstream release 8.6 for Pywikibot - https://phabricator.wikimedia.org/T354077 (10Xqt) [15:03:20] (HAProxyBackendUnavailable) resolved: HAProxy service nova-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [16:51:32] 10PAWS, 10Pywikibot: PAWS. pwb code entry point does not invoke scripts from scripts path - https://phabricator.wikimedia.org/T354084 (10Xqt) [17:05:03] (InstanceDown) firing: Project tools instance tools-harbor-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:10:03] (InstanceDown) resolved: Project tools instance tools-harbor-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:26:42] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [18:36:03] (InstanceDown) firing: Project tools instance tools-harbor-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:41:03] (InstanceDown) resolved: Project tools instance tools-harbor-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:05:03] (InstanceDown) firing: Project tools instance tools-harbor-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:20:03] (InstanceDown) resolved: Project tools instance tools-harbor-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:25:35] (HarborDown) firing: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [19:46:28] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [19:51:27] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [19:56:27] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:05:35] (HarborDown) resolved: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [21:47:56] (ToolsGridQueueProblem) firing: Grid queue webgrid-lighttpd@tools-sgeweblight-10-28.tools.eqiad1.wikimedia.cloud is in state E - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsGridQueueProblem - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsGridQueueProblem [22:46:27] (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse