[00:16:56] RESOLVED: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1002 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:19:26] RESOLVED: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1002. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:50:37] FIRING: [4x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:59:33] FIRING: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [01:29:40] (03update) 10raymond-ndibe: builds-api: bump to 0.0.212-20260428143311-7d2493f4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1237 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:29:44] (03approved) 10raymond-ndibe: builds-api: bump to 0.0.212-20260428143311-7d2493f4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1237 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:29:47] (03update) 10raymond-ndibe: builds-api: bump to 0.0.212-20260428143311-7d2493f4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1237 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:30:15] (03merge) 10raymond-ndibe: builds-api: bump to 0.0.212-20260428143311-7d2493f4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1237 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:33:49] (03update) 10raymond-ndibe: builds-builder: bump to 0.0.148-20260428143158-a5e19815 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1227 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:33:52] (03approved) 10raymond-ndibe: builds-builder: bump to 0.0.148-20260428143158-a5e19815 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1227 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:34:02] (03update) 10raymond-ndibe: builds-builder: bump to 0.0.148-20260428143158-a5e19815 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1227 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:34:09] (03update) 10raymond-ndibe: builds-builder: bump to 0.0.148-20260428143158-a5e19815 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1227 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:34:31] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.93-20260428143555-d3e3148d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1229 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:34:36] (03approved) 10raymond-ndibe: api-gateway: bump to 0.0.93-20260428143555-d3e3148d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1229 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:34:46] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.93-20260428143555-d3e3148d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1229 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:34:52] (03merge) 10raymond-ndibe: builds-builder: bump to 0.0.148-20260428143158-a5e19815 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1227 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:36:01] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.93-20260428143555-d3e3148d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1229 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:36:29] (03merge) 10raymond-ndibe: api-gateway: bump to 0.0.93-20260428143555-d3e3148d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1229 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:36:30] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.70-20260428143129-1fde83a4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1230 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [01:36:34] (03approved) 10raymond-ndibe: maintain-harbor: bump to 0.0.70-20260428143129-1fde83a4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1230 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [02:03:18] (03merge) 10raymond-ndibe: maintain-harbor: bump to 0.0.70-20260428143129-1fde83a4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1230 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [02:03:38] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.84-20260428143331-13957a07 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1234 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [02:03:43] (03approved) 10raymond-ndibe: envvars-api: bump to 0.0.84-20260428143331-13957a07 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1234 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [02:03:46] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.84-20260428143331-13957a07 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1234 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [02:04:11] (03merge) 10raymond-ndibe: envvars-api: bump to 0.0.84-20260428143331-13957a07 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1234 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [02:08:06] (03update) 10raymond-ndibe: upgrade python to 3.13 [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/91 (https://phabricator.wikimedia.org/T422184) [02:10:37] (03update) 10raymond-ndibe: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417 https://phabricator.wikimedia.org/T424362) (owner: 10fnegri) [02:50:21] (03update) 10raymond-ndibe: toolforge_get_versions.sh: remove ingress-nginx [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1226 [02:51:00] (03update) 10raymond-ndibe: tests: fix wrong test data [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/290 (https://phabricator.wikimedia.org/T423880) [02:54:49] (03CR) 10MusikAnimal: "For `/ec/en.wikisource.org/Alien333` I get:" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) (owner: 10Alien4444) [02:56:38] (03CR) 10MusikAnimal: [C:03+2] TopEdits: add back sha1-based revert detection [labs/xtools] - 10https://gerrit.wikimedia.org/r/1275237 (https://phabricator.wikimedia.org/T407814) (owner: 10Alien4444) [02:57:35] (03Merged) 10jenkins-bot: TopEdits: add back sha1-based revert detection [labs/xtools] - 10https://gerrit.wikimedia.org/r/1275237 (https://phabricator.wikimedia.org/T407814) (owner: 10Alien4444) [03:09:19] (03CR) 10MusikAnimal: [C:03+2] PageInfo: add message for < 30 page watchers (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1275239 (https://phabricator.wikimedia.org/T401360) (owner: 10Alien4444) [03:10:05] (03Merged) 10jenkins-bot: PageInfo: add message for < 30 page watchers [labs/xtools] - 10https://gerrit.wikimedia.org/r/1275239 (https://phabricator.wikimedia.org/T401360) (owner: 10Alien4444) [03:14:53] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 07Essential-Work, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)): Linter: Split rules into more logical groupings - https://phabricator.wikimedia.org/T415914#11869359 (10aaron) [04:18:18] (03update) 10raymond-ndibe: improve image parsing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/282 (https://phabricator.wikimedia.org/T415322) [04:19:40] (03update) 10raymond-ndibe: improve image parsing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/282 (https://phabricator.wikimedia.org/T415322) [04:20:52] (03update) 10raymond-ndibe: improve image parsing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/282 (https://phabricator.wikimedia.org/T415322) [05:47:22] (03update) 10isaranto: channels: update wikimedia-ml tags to match quarter variants [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/69 [06:39:43] 10Toolforge, 06tools-platform-team: Document updated process for cloud services survey - https://phabricator.wikimedia.org/T424609#11869547 (10komla) Wiki draft work happening here: [[https://wikitech.wikimedia.org/wiki/User:SSapaty_(WMF)/Draft/Annual_Cloud_Services_Survey]] [06:52:32] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work, 13Patch-For-Review: Add linting rules for operations and paths - https://phabricator.wikimedia.org/T422504#11869570 (10KineticPelagic) Of the comments I posted in my draft Merge R... [07:31:04] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Document linter rules - https://phabricator.wikimedia.org/T422930#11869650 (10KBach) 05Open→03In progress [08:03:05] 06cloud-services-team, 10Cloud-VPS: cloudvirt1075 in 'maintenance' aggregate - https://phabricator.wikimedia.org/T424802 (10fgiunchedi) 03NEW [08:53:52] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work: Add the remaining linting rules - https://phabricator.wikimedia.org/T422600#11869964 (10KBach) [08:59:30] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work: Add the remaining linting rules - https://phabricator.wikimedia.org/T422600#11869974 (10KBach) [09:24:36] 06Toolforge-standards-committee: Share/publish source code of "request" tool - https://phabricator.wikimedia.org/T416678#11870066 (10Tkarcher) *bump* If you find some time, could you please prioritize preparing and synching https://gitlab.wikimedia.org/toolforge-repos/request-review_bot ? There seems to be an i... [09:29:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [09:38:41] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:39:49] (03CR) 10Alien4444: PageInfo: add message for < 30 page watchers (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1275239 (https://phabricator.wikimedia.org/T401360) (owner: 10Alien4444) [09:49:34] RESOLVED: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [09:53:17] 06cloud-services-team: Make toolsbeta paging alerts less confusing - https://phabricator.wikimedia.org/T424814 (10fgiunchedi) 03NEW [09:53:40] 10Cloud-VPS, 06tools-infrastructure-team: Migrate Tools Platform Envoy TLS proxy services to the 2026 discovery intermediate - https://phabricator.wikimedia.org/T424675#11870186 (10MoritzMuehlenhoff) 05Open→03Resolved a:03taavi [09:53:45] 10Tool-Pageviews, 06Community-Tech (Fox Squad): [Cloud VPS alert][commtech] Puppet failure on pageviews02.commtech.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T424813#11870190 (10TheresNoTime) [09:54:09] 10Tool-Pageviews, 06Community-Tech (Fox Squad), 07Essential-Work: [Cloud VPS alert][commtech] Puppet failure on pageviews02.commtech.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T424813#11870192 (10TheresNoTime) [09:58:33] FIRING: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [10:04:39] (03PS9) 10Alien4444: Add proofreadpage quality support [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) [10:09:05] (03CR) 10Alien4444: "Hah! this is a funny "bug" if I've seen one. Essentially, the other day I was messing around with contentmodels for some test or other so " [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) (owner: 10Alien4444) [10:14:50] (03PS2) 10Alien4444: Use tabs not spaces in twig files and SQL queries [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276362 [10:15:40] 10Tool-Pageviews, 06Community-Tech (Fox Squad), 07Essential-Work: [Cloud VPS alert][commtech] Puppet failure on pageviews02.commtech.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T424813#11870294 (10TheresNoTime) `lang=bash samtar@pageviews02:~$ sudo puppet agent -tv Info: Using environment 'pro... [10:19:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [10:23:47] (03PS10) 10Alien4444: Add proofreadpage quality support [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) [10:23:55] (03CR) 10Alien4444: "You know what, on thinking further about it, it's a wiki we're talking about, and hardening never killed anyone, so I added a check defaul" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) (owner: 10Alien4444) [10:53:37] 06cloud-services-team, 10Cloud-VPS, 10Tool-Pageviews, 06Community-Tech (Fox Squad), 07Essential-Work: [Cloud VPS alert][commtech] Puppet failure on pageviews02.commtech.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T424813#11870388 (10TheresNoTime) [10:58:18] 06cloud-services-team, 10Cloud-VPS, 10Tool-Pageviews, 06Community-Tech (Fox Squad), 07Essential-Work: [Cloud VPS alert][commtech] Puppet failure on pageviews02.commtech.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T424813#11870434 (10TheresNoTime) (with thanks to @taavi for the suggestion)... [11:01:50] 06cloud-services-team, 10Cloud-VPS, 10Tool-Pageviews, 06Community-Tech (Fox Squad), 07Essential-Work: [Cloud VPS alert][commtech] Puppet failure on pageviews02.commtech.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T424813#11870450 (10TheresNoTime) should be fine now but I'll keep an eye out... [11:21:23] 10Cloud-VPS, 06tools-infrastructure-team: Migrate Tools Infrastructure Envoy TLS proxy services to the 2026 discovery intermediate - https://phabricator.wikimedia.org/T424675#11870575 (10taavi) [11:47:56] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: oslo.messaging does not failover to the next rabbit host on traffic blackhole situations - https://phabricator.wikimedia.org/T422820#11870726 (10fgiunchedi) Upstream bug at https://bugs.launchpad.net/oslo.messaging/+bug/2150632 [11:59:23] (03CR) 10A smart kitten: "I defer to you on this, but I wonder if it might be worth having a Phab task for a Hound upgrade; if only so that folks have somewhere to " [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1267343 (owner: 10Krinkle) [12:16:11] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 06Test Platform, 07OKR-Work: Prepare the linter for use in CI - https://phabricator.wikimedia.org/T422918#11870890 (10KBach) p:05Triage→03High [12:38:30] (03CR) 10A smart kitten: hound: Upgrade Hound from v0.4.0 to v0.7.1 (031 comment) [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1267343 (owner: 10Krinkle) [13:02:16] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 13Patch-For-Review: Replace ingress-nginx before upstream EOL date - https://phabricator.wikimedia.org/T392356#11871150 (10Yug) FYI, https://lingua-libre.toolforge.org likely for affected by this change, see : T424029. We fixed it by adding `buffer-siz... [13:37:26] (03update) 10isaranto: channels: update wikimedia-ml tags to match quarter variants [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/69 [13:38:56] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:42:15] 10Cloud-VPS (Project-requests): Request creation of wise VPS project - https://phabricator.wikimedia.org/T424068#11871325 (10Carandraug) It is possible to shard wise but if each shard would only have 4GB I don't think that's worth it. I didn't measure it but I wouldn't be surprised if there was at least 1GB of o... [13:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:53:03] (03approved) 10volans: Catch SQL errors [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/12 (https://phabricator.wikimedia.org/T351637) (owner: 10fnegri) [14:00:55] (03approved) 10aikochou: channels: update wikimedia-ml tags to match quarter variants [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/69 (owner: 10isaranto) [14:07:37] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#11871551 (10aputhin) [14:11:29] (03update) 10isaranto: channels: update wikimedia-ml tags to match quarter variants [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/69 [14:13:19] 10Toolforge, 06tools-platform-team: Continuous job failed to start due to missing envvar specified in secrets specification - https://phabricator.wikimedia.org/T422929#11871602 (10aputhin) p:05Triage→03Low We still need to work on a proper fix for this, but setting as "low priority" for now as there's an e... [14:16:34] 10Toolforge, 06tools-platform-team: [builds-builder] use yq instead of tomljson/jq/jsontoml - https://phabricator.wikimedia.org/T422691#11871612 (10aputhin) p:05Triage→03Low [14:22:01] 10Toolforge, 06tools-platform-team: maintain-dbusers should not pass sensitive data in command line parameters - https://phabricator.wikimedia.org/T424209#11871641 (10aputhin) p:05Triage→03Medium [14:24:22] 10Toolforge, 06tools-platform-team: maintain-dbusers should gracefully handle missing kubeconfig - https://phabricator.wikimedia.org/T424207#11871655 (10aputhin) p:05Triage→03Medium [14:31:02] (03update) 10isaranto: channels: update wikimedia-ml tags to match quarter variants [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/69 [14:31:07] 06cloud-services-team, 10Toolforge, 10Tools, 06Security-Team, and 4 others: events tool leaks database credentials via phpinfo() - https://phabricator.wikimedia.org/T423940#11871744 (10aputhin) As the issue is seemingly fixed, can we marked this as resolved? Or is there anything else we can do from the pla... [14:34:15] 06cloud-services-team, 10Toolforge, 10Tools, 06Security-Team, and 4 others: events tool leaks database credentials via phpinfo() - https://phabricator.wikimedia.org/T423940#11871799 (10sbassett) >>! In T423940#11871744, @aputhin wrote: > As the issue is seemingly fixed, can we marked this as resolved? Or i... [14:34:17] 10Toolforge, 06tools-platform-team: [builds-api] `step-fix-procfile-args-passing` does not support default Procfile entries - https://phabricator.wikimedia.org/T424191#11871803 (10aputhin) p:05Triage→03Medium [14:35:17] 06cloud-services-team, 10Toolforge, 10Tools, 06Security-Team, and 4 others: events tool leaks database credentials via phpinfo() - https://phabricator.wikimedia.org/T423940#11871819 (10sbassett) 05Open→03Resolved p:05Triage→03Medium a:03LucasWerkmeister [14:36:06] 06cloud-services-team, 10Toolforge: Buildservice for Rust fails - https://phabricator.wikimedia.org/T423675#11871828 (10aputhin) 05Open→03Resolved a:03aputhin Marking as resolved for now as it doesn't seem to re-occur. Please reopen if needed. [14:36:15] 06cloud-services-team, 10Toolforge: Buildservice for Rust fails - https://phabricator.wikimedia.org/T423675#11871832 (10aputhin) a:05aputhin→03None [14:52:20] (03update) 10kineticpelagic: Add rules for operations and paths (T422504) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/17 [14:53:42] 10Toolforge, 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#11871945 (10fnegri) [14:53:51] 10Toolforge, 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api] allow exposing continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#11871946 (10fnegri) [14:54:18] 10Toolforge, 06tools-platform-team: [jobs-cli] support publishing continuous job to the internet - https://phabricator.wikimedia.org/T423410#11871948 (10fnegri) [14:54:25] 10Toolforge, 06tools-platform-team: [jobs-api] support exposing continuous jobs to the internet - https://phabricator.wikimedia.org/T423408#11871949 (10fnegri) [15:03:55] (03update) 10fnegri: improve image parsing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/282 (https://phabricator.wikimedia.org/T415322) (owner: 10raymond-ndibe) [15:18:47] (03update) 10fnegri: improve image parsing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/282 (https://phabricator.wikimedia.org/T415322) (owner: 10raymond-ndibe) [15:26:49] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work: Add the remaining linting rules - https://phabricator.wikimedia.org/T422600#11872107 (10KineticPelagic) a:03KineticPelagic [15:28:29] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work, 13Patch-For-Review: Add linting rules for operations and paths - https://phabricator.wikimedia.org/T422504#11872121 (10aaron) 05Open→03In progress [15:31:25] (03update) 10raymond-ndibe: improve image parsing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/282 (https://phabricator.wikimedia.org/T415322) [15:34:08] (03update) 10raymond-ndibe: improve image parsing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/282 (https://phabricator.wikimedia.org/T415322) [15:41:36] (03approved) 10fnegri: improve image parsing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/282 (https://phabricator.wikimedia.org/T415322) (owner: 10raymond-ndibe) [15:41:42] (03update) 10fnegri: improve image parsing tests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/282 (https://phabricator.wikimedia.org/T415322) (owner: 10raymond-ndibe) [15:58:28] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 07Essential-Work, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)): Linter: Split rules into more logical groupings - https://phabricator.wikimedia.org/T415914#11872260 (10aaron) 05In progress→03Resolved [15:58:36] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)): Fix linter issues discovered during implementation of the OAD example - https://phabricator.wikimedia.org/T414974#11872262 (10aaron) 05Open→03Resolved [16:01:54] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 05MW-1.46-notes (1.46.0-wmf.26; 2026-04-28), 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)): OpenAPI linting: Add missing OpenAPI spec elements to Response Components - https://phabricator.wikimedia.org/T422739#11872276 (10aaron) 05Open... [16:02:40] (03update) 10raymond-ndibe: refactor image parsing and handling [repos/cloud/toolforge/jobs-api] (improve_image_parsing_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/273 (https://phabricator.wikimedia.org/T415322) [16:06:57] (03update) 10raymond-ndibe: refactor image parsing and handling [repos/cloud/toolforge/jobs-api] (improve_image_parsing_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/273 (https://phabricator.wikimedia.org/T415322) [16:07:00] (03update) 10raymond-ndibe: refactor image parsing and handling [repos/cloud/toolforge/jobs-api] (improve_image_parsing_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/273 (https://phabricator.wikimedia.org/T415322) [16:15:12] 06cloud-services-team, 10PAWS, 10Wikibase GraphQL, 10Wikidata: GraphQL frontend challenges: evaluate better integrations with PAWS (Jupyter) - https://phabricator.wikimedia.org/T422817#11872332 (10Ottomata) Hi! We aren't sure how Data-Engineering can help here. Untagging ourselves for now, but feel free t... [16:31:39] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Migrate our use of osbpo away from mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423598#11872449 (10Andrew) 05Open→03Resolved This is done, but we're now only providing flamingo and gazpacho, not epoxy or dalmatian. I've remov... [16:32:07] (03update) 10fnegri: refactor image parsing and handling [repos/cloud/toolforge/jobs-api] (improve_image_parsing_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/273 (https://phabricator.wikimedia.org/T415322) (owner: 10raymond-ndibe) [16:32:08] (03approved) 10fnegri: refactor image parsing and handling [repos/cloud/toolforge/jobs-api] (improve_image_parsing_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/273 (https://phabricator.wikimedia.org/T415322) (owner: 10raymond-ndibe) [17:39:51] 10Cloud-VPS (Project-requests): Request creation of wise VPS project - https://phabricator.wikimedia.org/T424068#11872763 (10taavi) +1 [18:28:55] 10Cloud-VPS (Project-requests): Request creation of wise VPS project - https://phabricator.wikimedia.org/T424068#11872947 (10Gopavasanth) [19:30:19] 10VPS-project-Codesearch, 06Wikidata Platform Team: request: index wikidata-platform gitlab repos - https://phabricator.wikimedia.org/T424896 (10gmodena) 03NEW [19:31:30] (03PS1) 10Gmodena: write_config: add wikidata platform repos [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1279462 (https://phabricator.wikimedia.org/T424896) [19:32:41] (03CR) 10CI reject: [V:04-1] write_config: add wikidata platform repos [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1279462 (https://phabricator.wikimedia.org/T424896) (owner: 10Gmodena) [19:40:17] (03PS2) 10Gmodena: write_config: add wikidata platform repos [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1279462 (https://phabricator.wikimedia.org/T424896) [20:21:56] FIRING: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [20:23:36] (03update) 10danyya: Draft: Country stats [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/20 [20:27:00] 10Tool-digitalt2commons, 10WMNO-General: Digitalt2Commons: Investigate possibility for adding a button to the tool in Digitalt Museum proper - https://phabricator.wikimedia.org/T403708#11873356 (10jhsoby-WMNO) 05Open→03Declined (Mass closing open WMNO Phabricator tasks, as we have switched to other met... [20:34:23] 10VPS-project-Phabricator: https.www.mobile.com - https://phabricator.wikimedia.org/T424902 (10Hr574380) 03NEW [20:35:18] 10Cloud-VPS (Project-requests): Request creation of wolf-a Cloud VPS project - https://phabricator.wikimedia.org/T424192#11873414 (10Aklapper) @Moh777tyjer3c: Could you please answer all questions in the last comments? Thanks in advance! [20:41:54] 10Tool-humaniki-2: Localize table texts - https://phabricator.wikimedia.org/T423787#11873433 (10Danya) [20:44:42] (03update) 10danyya: Draft: Country stats [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/20 [20:48:40] (03update) 10danyya: Country stats [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/20 [20:48:47] (03merge) 10danyya: Country stats [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/20 [20:50:40] 10Tool-humaniki-2: Use transactions for snapshots - https://phabricator.wikimedia.org/T424903 (10Danya) 03NEW [20:50:48] 10Tool-humaniki-2: Use transactions for snapshots - https://phabricator.wikimedia.org/T424903#11873465 (10Danya) p:05Triage→03Medium [20:50:57] 10Tool-humaniki-2: Localize table texts - https://phabricator.wikimedia.org/T423787#11873467 (10Danya) p:05Medium→03Low [20:51:06] 10Tool-humaniki-2: Choose licence for charts - https://phabricator.wikimedia.org/T422336#11873469 (10Danya) p:05Medium→03Low [20:51:15] 10Tool-humaniki-2: Reduce snapshots frequency - https://phabricator.wikimedia.org/T424277#11873471 (10Danya) p:05Medium→03Low [20:51:20] 10Tool-humaniki-2: List data sources - https://phabricator.wikimedia.org/T422332#11873473 (10Danya) p:05Medium→03Low [20:52:06] 10Tool-humaniki-2: [Countries view] Stats based on nationality - https://phabricator.wikimedia.org/T422341#11873476 (10Danya) 05In progress→03Resolved [21:05:39] (03open) 10aghirelli: feat(linter): unify and improve example validation rules (T423411) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/18 [21:07:44] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work, 13Patch-For-Review: Improve linting - unify example validation - https://phabricator.wikimedia.org/T423411#11873500 (10AGhirelli-WMF)