[00:02:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [00:07:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [00:46:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:45:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [02:50:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [03:22:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [03:27:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [03:46:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [05:23:30] (03CR) 10Gopavasanth: [C: 04-2] "This patch is not working, media player is throwing error - DEMUXER_ERROR_COULD_NOT_OPEN: FFmpegDeemucer: open context failed" [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/959795 (owner: 10Vibhoo24) [05:41:16] 10Tool-toolwatch: Improve the functionality of Toolwatch - https://phabricator.wikimedia.org/T353573 (10Aklapper) @Gopavasanth, @Soda: There is both https://github.com/gopavasanth/ToolWatch/issues and https://phabricator.wikimedia.org/tag/tool-toolwatch/ . Where is the one canonical issue tracker for this codeba... [05:43:29] 10Grid-Engine-to-K8s-Migration: Migrate mathis-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319880 (10Aklapper) @Mathis_Benguigui Hi, we expect tool maintainers to be subscribed to https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.org/ . The... [05:45:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [05:47:18] 10Tool-toolwatch: Improve the functionality of Toolwatch - https://phabricator.wikimedia.org/T353573 (10Reputation22) We will close the github issue @Aklapper, thanks :) [05:48:10] 10VPS-project-Codesearch, 10Browser-Support-Firefox: codesearch crashes firefox tabs - https://phabricator.wikimedia.org/T353480 (10Aklapper) 05Open→03Invalid Cannot reproduce with firefox-120.0.1; in any case I'd say that website content making the Firefox browser engine crash should be tracked in Mozilla... [05:50:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [06:24:54] (03CR) 10Gopavasanth: [C: 04-2] "Thanks Punith, resolved at https://gitlab.wikimedia.org/cloudvps-repos/videocuttool/VideoCutTool/-/commit/160898f7279a8a9790ea4b97682f4ebe" [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/892381 (https://phabricator.wikimedia.org/T330601) (owner: 10Punith.nyk) [06:44:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [06:46:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:49:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [07:01:20] 10Toolforge, 10Documentation: Django socialauth OAUTH login fails with mediawiki backend - https://phabricator.wikimedia.org/T353593 (10Zache) [07:28:05] (03Abandoned) 10Punith.nyk: Upgraded the Urlbox UI [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/945926 (owner: 10Punith.nyk) [07:29:03] 10Grid-Engine-to-K8s-Migration, 10User-Jayprakash12345: Migrate indic-wsstats from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319811 (10ParasharSarthak) GridEngine is used on indic-wsstats tool for cronjobs to update the stats. I have removed the all cronjobs and configur... [07:29:42] 10Grid-Engine-to-K8s-Migration, 10User-Jayprakash12345: Migrate indic-wsstats from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319811 (10ParasharSarthak) tools.indic-wsstats@tools-sgebastion-10:~$ toolforge jobs list Job name: Job type: Status: ----------- ---... [07:34:34] (03Abandoned) 10Punith.nyk: Allow video upload without modifications [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/890137 (https://phabricator.wikimedia.org/T283414) (owner: 10Punith.nyk) [07:34:56] (03Abandoned) 10Punith.nyk: Upgraded the footer UI [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/941777 (owner: 10Punith.nyk) [07:35:21] (03Abandoned) 10Punith.nyk: Upgraded the Header UI [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/941841 (owner: 10Punith.nyk) [07:35:29] 10Grid-Engine-to-K8s-Migration, 10User-Jayprakash12345: Migrate indic-wsstats from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319811 (10ParasharSarthak) 05Open→03Resolved [07:35:53] (03Abandoned) 10Punith.nyk: Prevent lintable/build errors from making into production [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/938384 (https://phabricator.wikimedia.org/T331364) (owner: 10Punith.nyk) [08:45:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [08:50:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [09:38:02] 10Tool-toolwatch: Sort tools based on tool Title - https://phabricator.wikimedia.org/T353579 (10Gopavasanth) [09:46:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:05:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [10:07:37] (CephSlowOps) firing: Ceph cluster in eqiad has 1 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [10:07:43] 10cloud-services-team: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T352570 (10phaultfinder) [10:10:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [10:12:37] (CephSlowOps) resolved: Ceph cluster in eqiad has 1 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [11:45:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [11:55:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [12:00:18] 10Tool-toolwatch: Improve the functionality of Toolwatch - https://phabricator.wikimedia.org/T353573 (10Reputation22) cc @Jnanaranjan_sahu [12:05:02] 10Tool-toolwatch: Improve the functionality of Toolwatch - https://phabricator.wikimedia.org/T353573 (10Reputation22) We plan to add a notification system to the authors once the tool gets unavailable. Lets discuss the task here before moving forward with any further implementations. [12:46:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:57:05] 10Grid-Engine-to-K8s-Migration: Migrate pibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319967 (10Mike_Peel) 05Open→03Resolved a:03Mike_Peel Many thanks @taavi - I managed to figure it all out today, particularly using that last link. It seems it's basically a c... [13:31:13] 10Tool-wlh: Add support for other wikis - https://phabricator.wikimedia.org/T353606 (10Jack_who_built_the_house) [13:52:18] 10Tools: Make the tool work on all wikis - https://phabricator.wikimedia.org/T353607 (10Jack_who_built_the_house) [13:52:47] 10Tools: Make the TemplateCount tool work on all wikis - https://phabricator.wikimedia.org/T353607 (10Jack_who_built_the_house) [13:59:08] 10Grid-Engine-to-K8s-Migration: Migrate qic from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319982 (10Mike_Peel) 05Open→03Resolved a:05Eatcha→03Mike_Peel Migrated. [14:45:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [14:55:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [15:46:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:19:01] 10Grid-Engine-to-K8s-Migration: Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320140 (10LucasWerkmeister) >>! In T320140#9395060, @LucasWerkmeister wrote: > - Build an image that contains //all// the needed software. Java, Node, Python 3, whatever el... [17:24:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:29:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:45:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [17:55:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [18:27:13] 10Grid-Engine-to-K8s-Migration: Migrate enboten from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319721 (10Lejonel) I disabled the cronjob scheduling tasks with jsub. Hopefully I can get something working using instructions at [[wikitech:Help:Toolforge/Running Pywikibot scri... [18:46:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:30:27] 10Toolforge, 10Documentation: Find and fix inaccuracies in https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Django_OAuth_tool - https://phabricator.wikimedia.org/T245683 (10Reedy) [20:31:19] 10Toolforge, 10Documentation: Django socialauth OAUTH login fails with mediawiki backend - https://phabricator.wikimedia.org/T353593 (10Reedy) [20:45:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [20:54:11] 10Toolforge-standards-committee: Adoption request for TOOLNAME - https://phabricator.wikimedia.org/T353609 (10DB111) [20:54:43] 10Toolforge-standards-committee: Adoption request for kmlexport - https://phabricator.wikimedia.org/T353609 (10DB111) [20:55:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [21:01:18] 10Toolforge-standards-committee: Adoption request for kmlexport - https://phabricator.wikimedia.org/T353609 (10JJMC89) [21:02:30] 10Toolforge-standards-committee: Adoption request for kmlexport - https://phabricator.wikimedia.org/T353609 (10JJMC89) You are required to notify all maintainers. Please link to the required talk page posts on Wikitech. [21:46:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [21:52:40] 10Toolforge-standards-committee: Adoption request for kmlexport - https://phabricator.wikimedia.org/T353609 (10bd808) In {T92963}, specifically T92963#2265237, @para refused to assign a software license to his source code implementing klmexport. [22:00:35] 10Toolforge-standards-committee: Adoption request for image-metadata-viewer - https://phabricator.wikimedia.org/T338558 (10taavi) The home directory of this tool is empty excluding auto-generated files so I believe this can go forward. [22:05:43] 10Toolforge-standards-committee: Adoption request for geograph2commons - https://phabricator.wikimedia.org/T345707 (10taavi) Only secret thing in the home directory seems to be `/data/project/geograph2commons/oauth.ini`, which has the credentials for the [[ https://meta.wikimedia.org/wiki/Special:OAuthListConsum... [22:08:05] 10Toolforge-standards-committee: Adoption request for kmlexport - https://phabricator.wikimedia.org/T353609 (10DB111) [22:09:36] 10Toolforge-standards-committee: Adoption request for kmlexport - https://phabricator.wikimedia.org/T353609 (10DB111) Will wait another two weeks for wikitech and then ping (or close) here, thank you! [22:10:01] 10Toolforge-standards-committee: Adoption request for kmlexport - https://phabricator.wikimedia.org/T353609 (10DB111) [22:10:47] 10Toolforge-standards-committee: Adoption request for kmlexport - https://phabricator.wikimedia.org/T353609 (10DB111) [23:32:19] (HAProxyBackendUnavailable) firing: HAProxy service nova-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [23:34:26] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [23:37:19] (HAProxyBackendUnavailable) resolved: HAProxy service nova-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [23:45:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [23:55:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed