[01:27:45] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate maps-osmdb from bullseye to .. - https://phabricator.wikimedia.org/T432955#12296058 (10komla) @saper any luck with progress on this? [01:29:22] 10Cloud-VPS (Debian Bullseye Deprecation), 10a Wikimedia CLI: Migrate mwcli away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435778#12296059 (10komla) @Addshore @jeena @Ollie.Shotton_WMDE have we made any progress with this? [01:38:10] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create a list of candidate tools for Push-to-Deploy adoption - https://phabricator.wikimedia.org/T437257 (10komla) 03NEW [01:39:13] 10Tool-wikinewsie: New Wikipedia articles should be their own event type on Wikinewsie - https://phabricator.wikimedia.org/T431879#12296075 (10Pharos) Given that much of this has been handled by T434836 now, I think a good marker of new-article status could just be a [[ https://commons.wikimedia.org/wiki/Categor... [02:08:08] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create a list of candidate tools for Push-to-Deploy adoption - https://phabricator.wikimedia.org/T437257#12296093 (10komla) [02:31:21] 10Tools, 07Chinese-Sites: talk-snapshots: tool status - https://phabricator.wikimedia.org/T437173#12296094 (10Shizhao) [03:13:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:18:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:25:41] 10Toolforge (Quota-requests), 06tools-infrastructure-team: Request increased quota for wiki-economics Toolforge tool - https://phabricator.wikimedia.org/T436614#12296111 (10komla) @fnegri, noted. I will work on this [03:36:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:41:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:09:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:14:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:15:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:20:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:50:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:55:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:47:20] 10Cloud-VPS (Quota-requests): Quota increase request for project deployment-prep - https://phabricator.wikimedia.org/T437117#12296247 (10fgiunchedi) +1 LGTM [06:47:58] 10Cloud-VPS (Project-requests), 06MediaWiki-Media-Platform-Team: Request creation of imagor VPS project - https://phabricator.wikimedia.org/T437088#12296249 (10fgiunchedi) +1 to imagor [08:00:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:20:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:22:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:27:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:28:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:31:30] (03update) 10dreamyshade: Shift focus to list-making [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/2 [08:31:35] (03close) 10dreamyshade: Shift focus to list-making [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/2 [08:33:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:57:44] 10Cloud-VPS, 10Data-Services, 06tools-infrastructure-team, 06Data-Platform-SRE: Export additional nfsd stats from /proc/fs/nfsd - https://phabricator.wikimedia.org/T436252#12296508 (10fgiunchedi) 05Openβ†’03Resolved a:03fgiunchedi Metrics are exported, I have uploaded a dashboard to get nfsd stats... [09:00:55] 10Cloud-VPS, 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team, and 4 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12296522 (10fgiunchedi) 05Openβ†’03Resolved a:03fgiunchedi Calling this one done, dumps-nfs.w.o is... [09:06:08] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, 07Incident Severity 3, 07Wikimedia-Incident: 2026-07-11 Incident - Toolforge haproxies flapping - https://phabricator.wikimedia.org/T431946#12296555 (10MLechvien-WMF) @dcaro @taavi could we update the description of this incident? [09:06:27] 10Tool-phab-ban, 10Toolforge, 06tools-platform-team, 07Sustainability (Incident Followup): Intermittent timeouts on Phab-ban tool - https://phabricator.wikimedia.org/T431967#12296557 (10MLechvien-WMF) [09:11:00] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 07Sustainability (Incident Followup): Ditch wmcs-specific alerts for systemdunitdown in favor of production alerts - https://phabricator.wikimedia.org/T428873#12296564 (10MLechvien-WMF) [09:11:01] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 07Sustainability (Incident Followup): mwopenstackclients.py reliability - https://phabricator.wikimedia.org/T429003#12296566 (10MLechvien-WMF) [09:11:02] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 07Sustainability (Incident Followup): Openstack backups postgres cursor log errors - https://phabricator.wikimedia.org/T428865#12296565 (10MLechvien-WMF) [09:11:03] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 07Sustainability (Incident Followup): wmcs-backup fails to create differential when the data changed too much and fails the whole run - https://phabricator.wikimedia.org/T428995#12296569 (10MLechvien-WMF) [09:11:09] 10Cloud-VPS, 06tools-infrastructure-team, 07Sustainability (Incident Followup): Cloud VPS cinder backy config should ensure a specific number of backups is kept - https://phabricator.wikimedia.org/T428897#12296573 (10MLechvien-WMF) [09:11:11] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 07Sustainability (Incident Followup): Add monitoring for backy2 openstack backups - https://phabricator.wikimedia.org/T428893#12296572 (10MLechvien-WMF) [09:11:12] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 07Sustainability (Incident Followup): wmcs-backup should not delete backups before taking the new one - https://phabricator.wikimedia.org/T429002#12296571 (10MLechvien-WMF) [09:11:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:16:37] 10Cloud-VPS, 06tools-infrastructure-team: More granular 'wmcs' clusters for better observability - https://phabricator.wikimedia.org/T437272 (10fgiunchedi) 03NEW [09:21:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:45:42] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: [toolsbeta] probe flapping on ipv6 only - https://phabricator.wikimedia.org/T426584#12296702 (10fgiunchedi) I have put in a silence for `ProbeDown` + toolsbeta-test-k8s-haproxy-7:443 [10:52:20] (03update) 10dcaro: api: add envvars to source build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/184 (https://phabricator.wikimedia.org/T434496) [10:53:42] (03update) 10dcaro: api: add envvars to source build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/184 (https://phabricator.wikimedia.org/T434496) [10:57:02] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 13Patch-For-Review: Add support for x1 and x4 sections on wiki replicas on the load balancer layer - https://phabricator.wikimedia.org/T409560#12296937 (10taavi) 05Stalledβ†’03Resolved a:03taavi To help task tracking I'm closing this an... [10:58:36] (03update) 10dcaro: api: add envvars to source build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/184 (https://phabricator.wikimedia.org/T434496) [11:17:49] (03PS1) 10Muehlenhoff: Remove obsolete sub secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1337896 [11:24:49] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [11:24:56] (03update) 10tlepage: refactor: Parse Loki result streams into events [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [11:25:20] (03update) 10tlepage: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) [11:26:19] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete sub secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1337896 (owner: 10Muehlenhoff) [11:27:51] (03PS1) 10Muehlenhoff: Stub secrets for new Trixie OpenLDAP roles [labs/private] - 10https://gerrit.wikimedia.org/r/1337901 (https://phabricator.wikimedia.org/T331699) [11:30:33] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Stub secrets for new Trixie OpenLDAP roles [labs/private] - 10https://gerrit.wikimedia.org/r/1337901 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [11:33:15] (03PS1) 10Muehlenhoff: Remove Hiera entries for hosts long decommissioned [labs/private] - 10https://gerrit.wikimedia.org/r/1337903 [11:35:40] (03PS1) 10Muehlenhoff: Remove mcrouter stub secrets for long-decommissioned mw baremetal hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1337905 [11:36:30] (03CR) 10Majavah: [C:03+1] Remove Hiera entries for hosts long decommissioned [labs/private] - 10https://gerrit.wikimedia.org/r/1337903 (owner: 10Muehlenhoff) [11:36:54] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [11:37:01] (03update) 10tlepage: refactor: Parse Loki result streams into events [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [11:37:13] (03update) 10tlepage: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) [11:41:38] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove Hiera entries for hosts long decommissioned [labs/private] - 10https://gerrit.wikimedia.org/r/1337903 (owner: 10Muehlenhoff) [11:50:18] (03PS2) 10Muehlenhoff: Remove mcrouter stub secrets for long-decommissioned mw baremetal hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1337905 [11:51:13] (03update) 10tlepage: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) [11:51:21] (03update) 10tlepage: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) [11:51:32] (03update) 10tlepage: refactor: Generalize job event models [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [12:13:16] (03update) 10dcaro: api: add envvars to source build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/184 (https://phabricator.wikimedia.org/T434496) [12:13:33] (03CR) 10ClΓ©ment Goubert: [C:03+1] Remove mcrouter stub secrets for long-decommissioned mw baremetal hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1337905 (owner: 10Muehlenhoff) [12:17:34] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove mcrouter stub secrets for long-decommissioned mw baremetal hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1337905 (owner: 10Muehlenhoff) [12:21:12] (03update) 10dcaro: api: add envvars to source build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/184 (https://phabricator.wikimedia.org/T434496) [12:39:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [12:44:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [12:52:48] 06tools-platform-team, 10Phabricator-Bot-Requests: Bot account for tools-platform team task flows - https://phabricator.wikimedia.org/T437284 (10dcaro) 03NEW [13:22:12] 06tools-platform-team, 10Phabricator-Bot-Requests: Bot account for tools-platform team task flows - https://phabricator.wikimedia.org/T437284#12297345 (10Aklapper) > An email address (which can be invalid but must be unique): dcaro@wikmimedia.org If the typo is not intentional (?), then this won't work //if/... [13:24:40] 06tools-platform-team, 10Phabricator-Bot-Requests: Bot account for tools-platform team task flows - https://phabricator.wikimedia.org/T437284#12297353 (10dcaro) >>! In T437284#12297345, @Aklapper wrote: >> An email address (which can be invalid but must be unique): dcaro@wikmimedia.org > > If the typo is not... [13:31:04] (03update) 10dcaro: models.py: expose state and webservice_defaults in images endpoint [repos/cloud/toolforge/jobs-api] (add_webservice_endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/347 (https://phabricator.wikimedia.org/T431679) (owner: 10raymond-ndibe) [13:56:04] (03open) 10komla: maintain-kubeusers: Increase wiki-economics quotas [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1414 (https://phabricator.wikimedia.org/T436614) [14:01:36] (03approved) 10dcaro: maintain-kubeusers: Increase wiki-economics quotas [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1414 (https://phabricator.wikimedia.org/T436614) (owner: 10komla) [14:04:42] (03merge) 10komla: maintain-kubeusers: Increase wiki-economics quotas [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1414 (https://phabricator.wikimedia.org/T436614) [14:13:45] !log komla@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T436614) [14:13:51] !log komla@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers (T436614) [14:13:51] T436614: Request increased quota for wiki-economics Toolforge tool - https://phabricator.wikimedia.org/T436614 [14:18:59] !log komla@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T436614) [14:19:04] T436614: Request increased quota for wiki-economics Toolforge tool - https://phabricator.wikimedia.org/T436614 [14:20:13] 06tools-platform-team, 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎): Bot account for tools-platform team task flows - https://phabricator.wikimedia.org/T437284#12297654 (10Aklapper) a:05dcaroβ†’03Aklapper Created https://phabricator.wikimedia.org/p/ToolsPlatformBot/ Its API Token is in... [14:23:38] 06tools-platform-team, 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎): Bot account for tools-platform team task flows - https://phabricator.wikimedia.org/T437284#12297665 (10ToolsPlatformBot) Test from the bot account [14:24:08] 06tools-platform-team, 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎): Bot account for tools-platform team task flows - https://phabricator.wikimedia.org/T437284#12297666 (10dcaro) 05Openβ†’03Resolved That worked :), closing, thanks a lot! [14:40:17] !log komla@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers (T436614) [14:40:22] T436614: Request increased quota for wiki-economics Toolforge tool - https://phabricator.wikimedia.org/T436614 [14:52:20] 10Toolforge (Quota-requests), 06tools-infrastructure-team: Request increased quota for wiki-economics Toolforge tool - https://phabricator.wikimedia.org/T436614#12297815 (10komla) 05Openβ†’03Resolved a:03komla This has been done: ` Running jobs Used Limit ----------... [15:29:17] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work: Create an extended property for UI friendly API module descriptions - https://phabricator.wikimedia.org/T434253#12298050 (10HCoplin-WMF) [15:41:07] (03update) 10dcaro: support --webservice option [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T428898) (owner: 10raymond-ndibe) [15:49:43] (03open) 10dcaro: package: explicitly mention the image when importing [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/99 [15:51:03] FIRING: ProjectProxyMainProxyDown: Proxy service address is unreachable - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyDown [15:51:44] (03update) 10dcaro: package: explicitly mention the image when importing [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/99 [15:52:01] (03update) 10dcaro: package: skip image on package:deb, as it's in the extended job [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/99 [15:53:20] (03update) 10dcaro: package: skip image on package:deb, as it's in the extended job [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/99 [15:56:03] RESOLVED: ProjectProxyMainProxyDown: Proxy service address is unreachable - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyDown [16:28:28] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate utrs away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435787#12298695 (10komla) @AmandaNP please let us know how it is going. [16:55:02] (03approved) 10tlepage: package: skip image on package:deb, as it's in the extended job [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/99 (owner: 10dcaro) [16:55:05] (03update) 10tlepage: package: skip image on package:deb, as it's in the extended job [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/99 (owner: 10dcaro) [16:56:37] 10Cloud Services Proposals, 06cloud-services-team, 06tools-platform-team: Decision request - Update and expand on coding standards for toolforge - https://phabricator.wikimedia.org/T408782#12298930 (10dcaro) Current working document https://docs.google.com/document/d/1yhPYcDmzO2aXIzK1d_G_7JQUjukGiis_zuLeJZvp... [16:57:41] (03merge) 10dcaro: package: skip image on package:deb, as it's in the extended job [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/99 [17:05:17] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [17:05:24] (03update) 10tlepage: refactor: Parse Loki result streams into events [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [17:05:26] (03update) 10tlepage: refactor: Parse Loki result streams into events [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [17:05:34] (03update) 10tlepage: refactor: Parse Loki result streams into events [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [18:08:51] 10VPS-project-Wikistats: Wikistats for wikipedias is not reachable - https://phabricator.wikimedia.org/T435959#12299347 (10Dzahn) I don't know anything since I was completely away from work for weeks. I see this now and it was working before and it's working now. :) [18:09:29] 10VPS-project-Wikistats: Wikistats for wikipedias is not reachable - https://phabricator.wikimedia.org/T435959#12299348 (10Dzahn) 05Openβ†’03Resolved a:03Dzahn maybe there was maintenance on cloud VPS or it got scraped and you fixed it by restarting httpd. thanks! [18:14:29] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [18:14:30] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [18:14:43] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [18:14:44] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [18:14:56] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [18:14:58] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [18:15:59] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [18:16:00] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [18:17:30] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [18:17:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [18:20:25] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate utrs away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435787#12299391 (10AmandaNP) 05Openβ†’03Resolved a:03AmandaNP All instances with the old OS are already deleted, i just did not have the opportunity to update this. [18:36:01] 10Tool-Pageviews: Add PetScan as a source for Massviews - https://phabricator.wikimedia.org/T437012#12299497 (10MusikAnimal) This has been released, but I still want to remove the requirement to enter a project. [18:38:03] FIRING: ProjectProxyMainProxyDown: Proxy service address is unreachable - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyDown [18:43:03] RESOLVED: ProjectProxyMainProxyDown: Proxy service address is unreachable - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyDown [19:13:50] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [19:13:57] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [19:14:04] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [19:14:58] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [19:15:36] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [19:31:00] 10Cloud-VPS, 06tools-infrastructure-team, 06All-and-every-Wikisource: wsexport traffic issues - https://phabricator.wikimedia.org/T437349 (10Andrew) 03NEW [19:37:08] 10Cloud-VPS, 06tools-infrastructure-team, 06All-and-every-Wikisource, 10WS Export: wsexport traffic issues - https://phabricator.wikimedia.org/T437349#12299844 (10MusikAnimal) Without having looked into anything specifically, I highly suspect issues are similar to {T430888} (apart from running out of disk... [19:38:02] 10Cloud-VPS, 06tools-infrastructure-team, 06All-and-every-Wikisource: cloud-vps haproxy port exhaustion - https://phabricator.wikimedia.org/T437350 (10Andrew) 03NEW [20:36:45] 10Cloud-VPS (Debian Bullseye Deprecation), 10SRE-swift-storage: Migrate swift away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435783#12300211 (10komla) @MatthewVernon any luck with progress on this? [20:39:35] 10Cloud-VPS (Debian Bullseye Deprecation), 03Readers Essential Work (Sunsetting and cleanup): Migrate reading-web-staging away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435780#12300226 (10komla) @Jdlrobson-WMF any luck with progress on this migration? [20:41:36] 10Cloud-VPS (Debian Bullseye Deprecation), 10Continuous-Integration-Infrastructure: Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592#12300231 (10komla) @Jdforrester-WMF any luck with progress on this? [20:43:04] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate etytree away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435770#12300232 (10komla) @Epantaleo any luck with progress on this? [23:18:51] (03open) 10mahveotm: Draft: WIP: Add Cinder volume types and QoS management [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/350 (https://phabricator.wikimedia.org/T410148) [23:35:16] (03update) 10mahveotm: Draft: WIP: Add Cinder volume types and QoS management [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/350 (https://phabricator.wikimedia.org/T410148) [23:42:57] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: tofu-infra: add cinder volume types - https://phabricator.wikimedia.org/T410148#12300924 (10Mahveotm) Hiya @fnegri, I’ve started a [WIP here](https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/3... [23:43:15] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: tofu-infra: add cinder volume types - https://phabricator.wikimedia.org/T410148#12300928 (10Mahveotm) 05Openβ†’03In progress