[00:47:56] (ToolsGridQueueProblem) firing: Grid queue webgrid-lighttpd@tools-sgeweblight-10-28.tools.eqiad1.wikimedia.cloud is in state E - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsGridQueueProblem - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsGridQueueProblem [00:57:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [01:02:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [03:47:56] (ToolsGridQueueProblem) firing: Grid queue webgrid-lighttpd@tools-sgeweblight-10-28.tools.eqiad1.wikimedia.cloud is in state E - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsGridQueueProblem - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsGridQueueProblem [04:18:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [04:23:20] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [04:35:30] 10Data-Services: Missing linktarget rows on wiki replicas - https://phabricator.wikimedia.org/T354089 (10Legoktm) [06:47:56] (ToolsGridQueueProblem) firing: Grid queue webgrid-lighttpd@tools-sgeweblight-10-28.tools.eqiad1.wikimedia.cloud is in state E - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsGridQueueProblem - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsGridQueueProblem [09:47:56] (ToolsGridQueueProblem) firing: Grid queue webgrid-lighttpd@tools-sgeweblight-10-28.tools.eqiad1.wikimedia.cloud is in state E - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsGridQueueProblem - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsGridQueueProblem [10:06:02] 10Data-Services: Missing linktarget rows on wiki replicas - https://phabricator.wikimedia.org/T354089 (10taavi) The views are identical on both clouddb1013 and clouddb1017 and match what's in Puppet: ` root@clouddb1013:s1[(none)]> show create table enwiki_p.linktarget\G *************************** 1. row *******... [11:01:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [11:06:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [11:16:36] 10Grid-Engine-to-K8s-Migration: Migrate phetools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319965 (10Xover) ##Quick status update The web front end uses both PHP and Python, so it can't run on any of the standard k8s images. The backend services are all written in P... [11:57:36] 10Grid-Engine-to-K8s-Migration: Migrate phetools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319965 (10Xover) ##Build Service image requirements From our custom Build Service image we're going to have to figure out how to meet at least the following requirements (in ca... [12:10:58] 10Tool-ldap: Fails with users that are not in any groups - https://phabricator.wikimedia.org/T354097 (10taavi) [12:32:38] 10Tool-ldap, 10Patch-For-Review: Fails with users that are not in any groups - https://phabricator.wikimedia.org/T354097 (10CodeReviewBot) taavi opened https://gitlab.wikimedia.org/toolforge-repos/ldap/-/merge_requests/3 Do not fail on users without groups [12:42:24] 10Grid-Engine-to-K8s-Migration, 10Toolforge Build Service: New upstream release 8.6 for Pywikibot - https://phabricator.wikimedia.org/T354077 (10Xqt) [12:47:56] (ToolsGridQueueProblem) firing: Grid queue webgrid-lighttpd@tools-sgeweblight-10-28.tools.eqiad1.wikimedia.cloud is in state E - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsGridQueueProblem - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsGridQueueProblem [13:07:53] 10Tool-sbdjwhsh: Delete https://sbdjwhsh.toolforge.org/ (HTTP 504) - https://phabricator.wikimedia.org/T315149 (10Aklapper) [14:23:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [14:28:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:47:56] (ToolsGridQueueProblem) firing: Grid queue webgrid-lighttpd@tools-sgeweblight-10-28.tools.eqiad1.wikimedia.cloud is in state E - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsGridQueueProblem - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsGridQueueProblem [16:58:56] 10Grid-Engine-to-K8s-Migration: Migrate smallem from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320048 (10Klein) 05Open→03Resolved a:03Klein All Smallem's tasks have been successfully migrated to Toolforge Kubernetes. [17:44:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:49:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:59:48] 10Tool-ldap: Fails with users that are not in any groups - https://phabricator.wikimedia.org/T354097 (10CodeReviewBot) legoktm merged https://gitlab.wikimedia.org/toolforge-repos/ldap/-/merge_requests/3 Do not fail on users without groups [18:18:23] 10Tool-ldap: Fails with users that are not in any groups - https://phabricator.wikimedia.org/T354097 (10Legoktm) Thanks! It's deploying right now. I audited the rest of the `.unwrap()`s, I see one other potential issue, which is a group that has no members. While unlikely, I'll add a fix for that. [18:19:13] 10Tool-ldap: Fails with users that are not in any groups - https://phabricator.wikimedia.org/T354097 (10Legoktm) 05Open→03Resolved a:03taavi https://ldap.toolforge.org/user/taavi-test-20230928-01 works now [18:27:08] 10Tool-ldap: Display group IDs - https://phabricator.wikimedia.org/T353311 (10Legoktm) a:03Legoktm [18:47:56] (ToolsGridQueueProblem) firing: Grid queue webgrid-lighttpd@tools-sgeweblight-10-28.tools.eqiad1.wikimedia.cloud is in state E - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsGridQueueProblem - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsGridQueueProblem [19:45:30] 10Tool-ldap: Display group IDs - https://phabricator.wikimedia.org/T353311 (10Legoktm) 05Open→03Resolved https://gitlab.wikimedia.org/toolforge-repos/ldap/-/commit/ddc7ce4cbfd1b34382b24d4f7dedee888025ad1c {F41640301} [19:46:37] 10Tool-ldap, 10LDAP, 10Security: Non-existent users in the archiva-deployers LDAP group - https://phabricator.wikimedia.org/T224110 (10Legoktm) [20:32:26] 10Toolforge: Toolforge: consider introducing some kind of CLI feedback reporting tool - https://phabricator.wikimedia.org/T332904 (10Aklapper) 05Open→03Declined Boldly declining after a potential feel-good "let the team brainstorm a bit without a followup concept" team offsite activity. [20:38:27] 10cloud-services-team, 10Tech-Docs-Team, 10Documentation: What are our documentation wikis for? - https://phabricator.wikimedia.org/T324210 (10Aklapper) 05Declined→03Open [20:38:38] 10cloud-services-team, 10Tech-Docs-Team, 10Documentation: What are our documentation wikis for? - https://phabricator.wikimedia.org/T324210 (10Aklapper) 05Open→03Declined I assume this task is either in scope for #tech-docs-team or should be declined if nobody plans to look into this topic. [21:47:56] (ToolsGridQueueProblem) firing: Grid queue webgrid-lighttpd@tools-sgeweblight-10-28.tools.eqiad1.wikimedia.cloud is in state E - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsGridQueueProblem - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsGridQueueProblem [21:59:42] 10Cloud-VPS, 10cloud-services-team: Malformed web requests from cloud-vps to wiki.lll.lu - https://phabricator.wikimedia.org/T354101 (10Andrew) [22:04:13] 10Cloud-VPS, 10cloud-services-team: Malformed web requests from cloud-vps to wiki.lll.lu - https://phabricator.wikimedia.org/T354101 (10Andrew) Responded with: ` Hello, Alain. That traffic is originating from Wikimedia Cloud Services, a public cloud used for both internal and volunteer-maintained services.... [22:12:33] 10Cloud-VPS, 10cloud-services-team: Malformed web requests from cloud-vps to wiki.lll.lu - https://phabricator.wikimedia.org/T354101 (10RhinosF1) > And what why is it probing for Git? (which we don't use) The request is for the the Siteinfo part of the API which displays some statistics & info about the site... [22:26:40] 10Cloud-VPS, 10cloud-services-team, 10Patch-For-Review: Malformed web requests from cloud-vps to wiki.lll.lu - https://phabricator.wikimedia.org/T354101 (10CodeReviewBot) rhinosf1 opened https://gitlab.wikimedia.org/cloudvps-repos/wikistats/-/merge_requests/7 Fix broken string variables [22:31:49] 10Grid-Engine-to-K8s-Migration, 10Toolforge Build Service: New upstream release 8.6 for Pywikibot - https://phabricator.wikimedia.org/T354077 (10bd808) For the shared pywikibot-buildservice image, @taavi wrote up his planned upgrade process at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Pywikibo... [22:31:59] 10Cloud-VPS, 10VPS-project-Wikistats, 10cloud-services-team, 10User-RhinosF1: Malformed web requests from cloud-vps to wiki.lll.lu - https://phabricator.wikimedia.org/T354101 (10RhinosF1) p:05Triage→03High a:03RhinosF1 [22:33:08] 10Cloud-VPS, 10VPS-project-Wikistats, 10cloud-services-team, 10User-RhinosF1: Malformed web requests from cloud-vps to wiki.lll.lu - https://phabricator.wikimedia.org/T354101 (10RhinosF1) And that's also one of my vps projects, will try and get the fix deployed in the morning. Feel free to reach out if hav... [22:41:48] 10Grid-Engine-to-K8s-Migration: Migrate unpatrollededitstats from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320105 (10bd808) 05Open→03Declined I disabled the tool per the comments in T314167#9426014 and T320105#9426016. It is unfortunate that the tool was lost due to l... [23:12:56] (ToolsGridQueueProblem) firing: (2) Grid queue webgrid-lighttpd@tools-sgeweblight-10-21.tools.eqiad1.wikimedia.cloud is in state E - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsGridQueueProblem - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsGridQueueProblem