[18:36:56] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40378908239 (https://github.com/cluebotng/component-configs/commits/2d1531d1cbfbfa07a816f43b825ad9d8214f9996) [18:36:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [18:40:22] 10Tool-itwiki-orfanizzabot: itWiki's orfanizzabot interrupted after Mon Mar 15 12:20:11 2021 for no reason - https://phabricator.wikimedia.org/T279452#12474184 (10valerio.bozzolan) [18:40:27] 10Tool-itwiki-orfanizzabot: Improve OrphanizerBot to handle disambiguation pages differently - https://phabricator.wikimedia.org/T320693#12474185 (10valerio.bozzolan) [18:40:34] 10Tool-itwiki-orfanizzabot: If the section "Voci correlate" becomes empty, remove that section - https://phabricator.wikimedia.org/T338084#12474186 (10valerio.bozzolan) [18:43:38] 10Tool-itwiki-orfanizzabot: Migrate itwiki's OrfanizzaBot from the Toolforge tool "itwiki" to a dedicated tool - https://phabricator.wikimedia.org/T442429 (10valerio.bozzolan) 03NEW p:05Triage→03High [18:49:56] FIRING: [5x] SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:38:45] 10Tool-wiki-ocr, 10All-and-every-Wikisource: OCR tool in Wikisource not working - https://phabricator.wikimedia.org/T442268#12474411 (10Dick_Bos) It's repaired again! Thanks. [20:15:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:16:32] (03update) 10renovatebot: Update python Docker tag to v3.15.1 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [20:16:33] (03update) 10renovatebot: Update python Docker tag to v3.15.2 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [20:20:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:22:49] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 100% full for project tools-logging - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [20:23:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [20:33:40] 10Data-Services, 06tools-platform-team: ERROR 1356 (HY000): View 'commonswiki_p.globalimagelinks' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them - https://phabricator.wikimedia.org/T442452 (10grin) 03NEW [20:35:26] 10Data-Services, 06tools-platform-team: ERROR 1356 (HY000): View 'commonswiki_p.globalimagelinks' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them - https://phabricator.wikimedia.org/T442452#12474837 (10JJMC89) [20:39:23] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-37 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [20:40:17] 10Tool-itwiki-orfanizzabot: Migrate itwiki's OrfanizzaBot from the Toolforge tool "itwiki" to a dedicated tool - https://phabricator.wikimedia.org/T442429#12474858 (10valerio.bozzolan) Updated bot password. It works locally: https://it.wikipedia.org/w/index.php?title=Pyramid_Head&diff=prev&oldid=153081396 Depl... [20:41:11] (03update) 10zhaofjx: add support for URL parameter to change user interface using ?lang [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/31 (owner: 10chimaifeanyi29) [20:41:11] (03update) 10zhaofjx: add support for URL parameter to change user interface using ?lang [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/31 (owner: 10chimaifeanyi29) [20:41:34] 10Tool-itwiki-orfanizzabot: Migrate itwiki's OrfanizzaBot from the Toolforge tool "itwiki" to a dedicated tool - https://phabricator.wikimedia.org/T442429#12474862 (10valerio.bozzolan) Technical reference: https://wikitech.wikimedia.org/wiki/Tool:Itwiki-orfanizzabot [20:41:50] (03approved) 10zhaofjx: add support for URL parameter to change user interface using ?lang [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/31 (owner: 10chimaifeanyi29) [20:42:00] (03update) 10zhaofjx: add support for URL parameter to change user interface using ?lang [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/31 (owner: 10chimaifeanyi29) [20:42:21] (03merge) 10zhaofjx: add support for URL parameter to change user interface using ?lang [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/31 (owner: 10chimaifeanyi29) [20:45:08] 10Data-Services, 06tools-platform-team: ERROR 1356 (HY000): View 'commonswiki_p.globalimagelinks' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them - https://phabricator.wikimedia.org/T442452#12474872 (10grin) Affected on both *.web.db.svc.wikimedia.clou... [20:46:35] 10Tool-wikinewsie, 07good first task: Specify Wikinewsie interface language in url - https://phabricator.wikimedia.org/T441366#12474875 (10ZhaoFJx) 05In progress→03Resolved [20:47:32] 10Tool-itwiki-orfanizzabot: Migrate itwiki's OrfanizzaBot from the Toolforge tool "itwiki" to a dedicated tool - https://phabricator.wikimedia.org/T442429#12474886 (10valerio.bozzolan) 05Open→03Resolved Updated https://it.wikipedia.org/wiki/Utente:OrfanizzaBot#Documentazione_tecnica [20:49:23] RESOLVED: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-37 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [20:55:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:59:23] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-37 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [21:00:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:02:49] RESOLVED: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 100% full for project tools-logging - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [21:06:56] (03open) 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7: Bir ilçedeki aynı türden anıtları birlikte öner [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/7 [21:08:56] 10Tool-itwiki: Fix + harden itwiki tools after phpocalypse (October 2026) - https://phabricator.wikimedia.org/T442455 (10valerio.bozzolan) 03NEW p:05Triage→03High [21:09:52] 10Tool-itwiki-orfanizzabot: Migrate itwiki's OrfanizzaBot from the Toolforge tool "itwiki" to a dedicated tool - https://phabricator.wikimedia.org/T442429#12474957 (10valerio.bozzolan) [21:09:53] 10Tool-itwiki-bot-cancellazioni: Migrate itwiki BotCancellazioni from tool itwiki to dedicated tool - https://phabricator.wikimedia.org/T442396#12474958 (10valerio.bozzolan) [21:09:54] 10Tool-itwiki: Fix + harden itwiki tools after phpocalypse (October 2026) - https://phabricator.wikimedia.org/T442455#12474956 (10valerio.bozzolan) [21:10:04] 10Tool-itwiki: Fix + harden itwiki tools after phpocalypse (October 2026) - https://phabricator.wikimedia.org/T442455#12474960 (10valerio.bozzolan) [21:10:14] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: 2026-10-21 Toolforge and Cloud VPS down - https://phabricator.wikimedia.org/T441908#12474959 (10valerio.bozzolan) [21:13:34] 10Tool-itwiki: Fix + harden itwiki tools after phpocalypse (October 2026) - https://phabricator.wikimedia.org/T442455#12474963 (10valerio.bozzolan) [21:16:27] (03update) 10renovatebot: Update python Docker tag to v3.15.2 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [21:24:23] RESOLVED: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-37 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [21:38:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:50:25] 10Data-Services, 06tools-platform-team: ERROR 1356 (HY000): View 'commonswiki_p.globalimagelinks' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them - https://phabricator.wikimedia.org/T442452#12475093 (10Bawolff) Isnt this expected if you aren't using th... [22:47:32] (03open) 10mildrette: Convert the top viewed and the downloaded book tables into galleries T442168 [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/15 [22:49:56] FIRING: [5x] SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:56:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:01:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:28:01] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [23:29:08] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40419679333 (https://github.com/cluebotng/component-configs/commits/c35c9753fc0f5978462a889014ba06ed450c3e0a) [23:29:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [23:29:29] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40419679243 (https://github.com/cluebotng/component-configs/commits/c35c9753fc0f5978462a889014ba06ed450c3e0a) [23:29:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [23:41:18] 06Toolforge-standards-committee: November 2026 TFSC appointments - https://phabricator.wikimedia.org/T442469 (10JJMC89) 03NEW p:05Triage→03Medium