[00:29:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:34:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:36:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:41:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:52:08] 10Tool-wd-act, 07Security: Replace development OAuth token config with production config - https://phabricator.wikimedia.org/T431026#12214833 (10Tekneos) Based on the below guide, I think the best/most standard way to store the API credentials for the application, would be inside a file in the project's data d... [02:56:00] 10Tool-wd-act, 07Security: Replace development OAuth token config with production config - https://phabricator.wikimedia.org/T431026#12214836 (10Tekneos) On second thought, the scratch directory isn't a good idea. According to the below page, it's shared across all instances in all projects. [02:56:06] 10Tool-wd-act, 07Security: Replace development OAuth token config with production config - https://phabricator.wikimedia.org/T431026#12214837 (10Tekneos) https://wikitech.wikimedia.org/wiki/Help:Shared_storage#/data/scratch [03:06:25] 10Tool-wd-act, 07Security: Replace development OAuth token config with production config - https://phabricator.wikimedia.org/T431026#12214846 (10Tekneos) On third thought: Running this web app inside of kubernetes complicates things, because my image won't have access to the shared storage locations out of the... [04:07:44] 10Tool-wd-act: Accessing Q1299 returns an Internal Server Error - https://phabricator.wikimedia.org/T434858 (10Tekneos) 03NEW [04:28:31] 10Tool-wd-act, 07Security: Replace development OAuth token config with production config - https://phabricator.wikimedia.org/T431026#12214897 (10Tekneos) The production OAuth consumer is setup here: https://meta.wikimedia.org/wiki/Special:OAuthListConsumers/view/1ef316396fb537bfdd76a0c1051d2e98 The code for th... [04:38:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:43:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:53:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:29:48] (03update) 10renovatebot: Update python Docker tag to v3.14.7 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [05:40:46] (03update) 10countcount: Draft: Expose account block behavior flags [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/73 [06:13:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [07:39:24] (03approved) 10devjadiya: Remember last-selected license across uploads [toolforge-repos/clip2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/merge_requests/11 (https://phabricator.wikimedia.org/T434084) (owner: 10anushka10patel) [07:39:29] (03merge) 10devjadiya: Remember last-selected license across uploads [toolforge-repos/clip2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/merge_requests/11 (https://phabricator.wikimedia.org/T434084) (owner: 10anushka10patel) [07:40:53] 10Tool-clip2commons, 13Patch-For-Review: Remember previously selected license - https://phabricator.wikimedia.org/T434084#12215048 (10Dev) Completed and [[ https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/merge_requests/11 | merged ]]. License preference persistence is working as expected. Closed. [07:41:04] 10Tool-clip2commons, 13Patch-For-Review: Remember previously selected license - https://phabricator.wikimedia.org/T434084#12215049 (10Dev) 05Open→03Resolved [08:23:31] (03update) 10countcount: Draft: Expose account block behavior flags [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/73 [08:24:31] (03update) 10countcount: Draft: Expose account block behavior and historic restriction scopes [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/73 [08:44:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:48:36] (03update) 10danyya: Draft: Resolve T434400 "Dumps ingester crashes because of too many prepared statements (2)" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/109 [08:49:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:34:03] 10Tool-itwiki, 10MediaWiki-Action-API, 06MediaWiki-API-Platform-Team, 05MW-1.46-notes, 05MW-1.47-notes (1.47.0-wmf.15; 2026-08-11): Missing categories from generator=categorymembers (just 1 instead of 2) in itwiki - https://phabricator.wikimedia.org/T433922#12215332 (10Zabe) 05Open→03Resolved a:... [09:43:53] 06cloud-services-team, 10Data-Services, 06tools-platform-team: Deprecate /data/scratch volume - https://phabricator.wikimedia.org/T415819#12215340 (10taavi) [[ https://wikitech.wikimedia.org/w/index.php?diff=2447575 | Documented ]] that this volume will not be added to any new projects. [09:45:59] 10Data-Services, 06tools-infrastructure-team: Deprecate /data/scratch volume - https://phabricator.wikimedia.org/T415819#12215343 (10taavi) [09:47:33] (03open) 10filippo: rotate_clinic_duty: find last Thursday [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/9 [09:48:03] (03update) 10filippo: rotate_clinic_duty: find last Thursday [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/9 [09:48:05] 10Data-Services, 06tools-infrastructure-team, 10VPS-Projects: NFS shared storage for Cloud VPS project `globaleducation` - https://phabricator.wikimedia.org/T434723#12215348 (10taavi) What Bryan suggests is what I would do as well. Alternatively I'd recommend updating the jobs to automatically use `rsync` (o... [09:48:21] (03update) 10filippo: rotate_clinic_duty: find last Thursday [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/9 [09:48:56] (03update) 10filippo: rotate_clinic_duty: find last Thursday [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/9 [09:53:39] (03update) 10filippo: rotate_clinic_duty: find last Thursday [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/9 [09:58:57] 06cloud-services-team, 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431055#12215371 (10fgiunchedi) PAWS PR https://github.com/toolforge/paws/pull/529 [10:00:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:05:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:06:18] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7, tools-k8s-gateway-1 [10:08:02] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7, tools-k8s-gateway-1 [10:09:05] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8, tools-k8s-gateway-2 [10:09:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:10:14] FIRING: [4x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [10:10:42] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8, tools-k8s-gateway-2 [10:11:41] (03close) 10filippo: rotate_clinic_duty: find last Thursday [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/9 [10:12:50] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-gateway-3 [10:14:29] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-gateway-3 [10:14:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:15:14] RESOLVED: [11x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [10:16:14] FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-3.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [10:21:14] RESOLVED: [12x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [10:25:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:29:04] 06cloud-services-team, 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431055#12215436 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi {{done}} [10:30:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:45:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:02:36] (03update) 10tlepage: auth: Allow specifying allowed urls for superusers [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/103 (owner: 10dcaro) [11:02:54] (03update) 10tlepage: auth: Allow specifying allowed urls for superusers [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/103 (owner: 10dcaro) [11:53:26] !log taavi@cloudcumin1001 cloudinfra-nfs START - Cookbook wmcs.vps.add_user_to_project for user 'taavi' in role 'member' [11:53:31] !log taavi@cloudcumin1001 cloudinfra-nfs END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'taavi' in role 'member' [11:54:44] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Shutdown timeline extension requests for maintained projects - https://phabricator.wikimedia.org/T434103#12215640 (10taavi) [12:06:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:11:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:44:22] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: accounts.wmflabs.org is unreachable - https://phabricator.wikimedia.org/T434887 (10Xaosflux) 03NEW [12:49:45] 10Cloud-VPS, 06tools-infrastructure-team: accounts.wmflabs.org is unreachable - https://phabricator.wikimedia.org/T434887#12215858 (10taavi) 05Open→03Invalid > Please contact a maintainer of this project. Cloud VPS admins are not maintainers of the ACC project. I see this has already been reported to... [13:12:42] 10Data-Services, 06tools-infrastructure-team, 10VPS-Projects: NFS shared storage for Cloud VPS project `globaleducation` - https://phabricator.wikimedia.org/T434723#12215985 (10Gabinaluz) Thank you both for your help! I was reading S3-compatible object storage service and that might do the trick. [13:13:15] 10Data-Services, 06tools-infrastructure-team, 10VPS-Projects: NFS shared storage for Cloud VPS project `globaleducation` - https://phabricator.wikimedia.org/T434723#12215990 (10Gabinaluz) 05Open→03Resolved [13:14:42] 10Tool-wikinewsie: Wikinewsie cards to be sorted by type based on Wikidta - https://phabricator.wikimedia.org/T434836#12216022 (10ZhaoFJx) p:05Triage→03Medium [13:14:58] 10Tool-wikinewsie: Wikinewsie cards to be sorted by type based on Wikidta - https://phabricator.wikimedia.org/T434836#12216025 (10ZhaoFJx) 05Open→03In progress [13:54:08] 10Striker, 06tools-platform-team: 500 Internal Server Error when trying to log in to toolsadmin - https://phabricator.wikimedia.org/T434904 (10FlammablePizza) 03NEW [14:05:17] 10Toolforge, 06tools-platform-team: Establish design principles for parity between interfaces (CLI vs REST API) - https://phabricator.wikimedia.org/T434093#12216289 (10CCiufo-WMF) a:03CCiufo-WMF [14:05:23] 10Toolforge, 06tools-platform-team: Establish design principles for parity between interfaces (CLI vs REST API) - https://phabricator.wikimedia.org/T434093#12216290 (10CCiufo-WMF) 05Open→03In progress [14:07:34] 10Toolforge, 06tools-platform-team: Establish design principles for parity between interfaces (CLI vs REST API) - https://phabricator.wikimedia.org/T434093#12216293 (10CCiufo-WMF) The #tools-platform-team met yesterday for an initial discussion on potential design principles. Notes are available in [[ https://... [14:43:14] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12216514 (10Andrew) Another bookworm case: libup-runner08.library-upgrader.eqiad1.wikimedia.cloud [14:54:09] 10Cloud-VPS, 06tools-infrastructure-team: Improve VM backup performance - https://phabricator.wikimedia.org/T434014#12216534 (10Andrew) Both servers (1003 and 1004) completed their last runs in under 19 hours. That's not fantastic but it is fast enough to allow for proper cleanup between runs. [14:58:14] 10VPS-project-devtools, 10VPS-project-Phabricator, 06Collaboration-Services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12216606 (10Arnoldokoth) Trixie host now running under `phabricator.wmcloud... [14:58:23] 10VPS-project-devtools, 10VPS-project-Phabricator, 06Collaboration-Services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12216607 (10Arnoldokoth) 05Open→03Resolved [15:03:47] 10Cloud-VPS (Debian Bullseye Deprecation): Shutdown timeline extension requests for maintained projects - https://phabricator.wikimedia.org/T434103#12216650 (10taavi) [15:03:53] 10Cloud-VPS (Debian Bullseye Deprecation): Shutdown timeline extension requests for maintained projects - https://phabricator.wikimedia.org/T434103#12216654 (10taavi) [15:04:15] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12216655 (10taavi) [15:19:57] 10Data-Services, 06tools-infrastructure-team, 10VPS-Projects: NFS shared storage for Cloud VPS project `globaleducation` - https://phabricator.wikimedia.org/T434723#12216717 (10JJMC89) 05Resolved→03Declined [15:39:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:44:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:45:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:55:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:10:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:26:39] 06cloud-services-team, 10Tool-paulina, 10Toolforge, 06tools-platform-team: Automated traffic affects tool performance - https://phabricator.wikimedia.org/T432878#12217030 (10Pepe_piton) > In T432878#12210217, @bd808 wrote: > The more I look at this, I think moving to a custom build service container and di... [16:35:10] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12217066 (10fnegri) > we should at least be storing them so past alerts can be eventually retrieved. a.k.a. let's not lose data from this period pre-API. I discussed this w... [16:59:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:04:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:10:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:15:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:42:45] 10Tool-wikinewsie: Strip all Wikinewsie text to i18n files - https://phabricator.wikimedia.org/T434617#12217260 (10ZhaoFJx) 05In progress→03Resolved [18:10:24] 10Tool-wikimedia-attribution, 06MediaWiki-API-Platform-Team, 10MediaWiki-REST-API, 07OKR-Work: Include contributor_count in Attribution API responses - https://phabricator.wikimedia.org/T429834#12217318 (10HCoplin-WMF) a:03pmiazga @pmiazga -- for now, let's just include the raw numbers instead of working... [18:17:11] 10Tool-phab-ban, 10Toolforge, 06tools-platform-team: Intermittent timeouts on Phab-ban tool - https://phabricator.wikimedia.org/T431967#12217332 (10Aklapper) [18:28:24] (03open) 10ranjithsiji: perf: speed up gap analysis via replica queries and parallelism [toolforge-repos/wikiget] - 10https://gitlab.wikimedia.org/toolforge-repos/wikiget/-/merge_requests/1 [18:28:32] (03approved) 10ranjithsiji: perf: speed up gap analysis via replica queries and parallelism [toolforge-repos/wikiget] - 10https://gitlab.wikimedia.org/toolforge-repos/wikiget/-/merge_requests/1 [18:28:40] (03merge) 10ranjithsiji: perf: speed up gap analysis via replica queries and parallelism [toolforge-repos/wikiget] - 10https://gitlab.wikimedia.org/toolforge-repos/wikiget/-/merge_requests/1 [19:41:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:49:11] (03open) 10ranjithsiji: feat: save translated labels and descriptions to Wikidata [toolforge-repos/wikiget] - 10https://gitlab.wikimedia.org/toolforge-repos/wikiget/-/merge_requests/2 [19:49:16] (03approved) 10ranjithsiji: feat: save translated labels and descriptions to Wikidata [toolforge-repos/wikiget] - 10https://gitlab.wikimedia.org/toolforge-repos/wikiget/-/merge_requests/2 [19:49:19] (03merge) 10ranjithsiji: feat: save translated labels and descriptions to Wikidata [toolforge-repos/wikiget] - 10https://gitlab.wikimedia.org/toolforge-repos/wikiget/-/merge_requests/2 [19:51:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:28:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:33:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:38:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:44:32] 10Tool-wikinewsie: Wikinewsie 'Mix' interface takes up too much space on mobile - https://phabricator.wikimedia.org/T434606#12217727 (10ZhaoFJx) 05In progress→03Resolved [21:38:37] 10Tool-itwiki: BotCancellazioni: rollback "cllimit=500" workaround after T433922 has been resolved - https://phabricator.wikimedia.org/T434598#12218011 (10Umherirrender) Each request consumes resources, there are checks for each request, for example if allowed to do, if logged in, logging etc. So when doing less... [22:35:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown