[08:10:46] 06serviceops, 06Growth-Team, 06MediaWiki-Engineering, 10MediaWiki-Maintenance-system, and 2 others: mwscript-k8s makes it challenging to run LoggedUpdateMaintenance across all wikis - https://phabricator.wikimedia.org/T411104#11431652 (10daniel) a:03daniel [08:17:50] 06serviceops, 06Growth-Team, 06MediaWiki-Engineering, 10MediaWiki-Maintenance-system, and 2 others: mwscript-k8s makes it challenging to run LoggedUpdateMaintenance across all wikis - https://phabricator.wikimedia.org/T411104#11431680 (10daniel) I made a patch for core and for the two relevant maintenance... [10:57:46] 06serviceops, 06SRE Observability, 06MediaWiki-Platform-Team (Kanban Board): MediaWiki periodic job startupregistrystats-mediawikiwiki failed - https://phabricator.wikimedia.org/T410764#11432069 (10Clement_Goubert) >>! In T410764#11429889, @Tgr wrote: > In general I think it would be nice to rename these aut... [12:07:24] 06serviceops, 06Growth-Team, 10PageViewInfo, 06Content-Transform-Team (Work In Progress), and 3 others: Determine the source of internal requests going through the API gateway. - https://phabricator.wikimedia.org/T410198#11432298 (10Clement_Goubert) [12:13:49] 06serviceops, 06Growth-Team, 10PageViewInfo, 06Content-Transform-Team (Work In Progress), and 3 others: Determine the source of internal requests going through the API gateway. - https://phabricator.wikimedia.org/T410198#11432342 (10Clement_Goubert) [12:15:00] 06serviceops, 06SRE Observability, 06MediaWiki-Platform-Team (Kanban Board): MediaWiki periodic job startupregistrystats-mediawikiwiki failed - https://phabricator.wikimedia.org/T410764#11432344 (10Tgr) >>! In T410764#11432069, @Clement_Goubert wrote: > If you rename them, the next time the alert fires, it w... [12:24:03] 06serviceops, 06SRE Observability, 06MediaWiki-Platform-Team (Kanban Board): MediaWiki periodic job startupregistrystats-mediawikiwiki failed - https://phabricator.wikimedia.org/T410764#11432368 (10Tgr) The dashboard URL structure is pretty obscure so I [[https://wikitech.wikimedia.org/wiki/OpenSearch_Dashbo... [12:39:24] 06serviceops, 06Growth-Team, 10PageViewInfo, 06Content-Transform-Team (Work In Progress), and 3 others: Determine the source of internal requests going through the API gateway. - https://phabricator.wikimedia.org/T410198#11432400 (10Clement_Goubert) As far as I can tell from [[ https://logstash.wikimedia.o... [13:02:09] 06serviceops, 06SRE Observability, 06MediaWiki-Platform-Team (Kanban Board): MediaWiki periodic job startupregistrystats-mediawikiwiki failed - https://phabricator.wikimedia.org/T410764#11432446 (10Clement_Goubert) >>! In T410764#11432344, @Tgr wrote: >>>! In T410764#11432069, @Clement_Goubert wrote: >> If y... [13:11:21] 06serviceops, 06Growth-Team, 10PageViewInfo, 06Content-Transform-Team (Work In Progress), and 3 others: Determine the source of internal requests going through the API gateway. - https://phabricator.wikimedia.org/T410198#11432458 (10Clement_Goubert) [13:14:06] 06serviceops, 07Documentation, 05WE4.2 Bot detection: hcaptcha proxy: update wikitech page - https://phabricator.wikimedia.org/T411131#11432486 (10Raine) [13:53:48] 06serviceops: hcaptcha-proxy: create logstash dashboard - https://phabricator.wikimedia.org/T411780 (10Raine) 03NEW [13:57:52] 06serviceops: hcaptcha-proxy: consider unit tests, integration tests - https://phabricator.wikimedia.org/T411782 (10Raine) 03NEW [13:58:34] 06serviceops, 10GitLab (CI & Job Runners): failed to push docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.10.5-py3.11-2025-10-03-192132-3003d4328df66a0086a350fdd2ba1dbd80a235c5: unknown: blob upload invalid - https://phabricator.wikimedia.org/T406392#11432657 (10BTullis) In case i... [14:04:59] 06serviceops, 07Documentation, 05WE4.2 Bot detection: hcaptcha proxy: update wikitech page - https://phabricator.wikimedia.org/T411131#11432688 (10Raine) [15:00:06] 06serviceops: Monitor hCaptcha (upstream) status - https://phabricator.wikimedia.org/T411255#11433013 (10Raine) [16:17:40] 06serviceops: WE6.2.6: ☂️ hcaptcha-proxy Production Readiness Review - https://phabricator.wikimedia.org/T410626#11433310 (10jijiki) [16:25:23] 06serviceops, 10GitLab (CI & Job Runners): failed to push docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.10.5-py3.11-2025-10-03-192132-3003d4328df66a0086a350fdd2ba1dbd80a235c5: unknown: blob upload invalid - https://phabricator.wikimedia.org/T406392#11433342 (10elukey) >>! In T40... [16:28:26] 06serviceops, 13Patch-For-Review: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#11433354 (10elukey) My understanding from the last updates is that we are not actively pushing anything to the new registry with apus as backend, we have just quickly tested some... [16:32:34] 06serviceops, 13Patch-For-Review: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#11433382 (10MatthewVernon) per ` radosgw-admin user stats --uid=docker-registry`there are only 11 objects in that account, which I think equates to it not being currently used. [16:51:25] 06serviceops, 06Commons, 10Thumbor, 07Wikimedia-production-error: No thumbnail available for File:Himalaya,_Indian_Atlas,_sheet_66_(15219000).jpg - Error: 429, Too Many Requests - https://phabricator.wikimedia.org/T392348#11433459 (10Yann) I think I found a way to fix that. Saving in with baseline inst... [17:58:42] 06serviceops, 10Wikifunctions, 06Abstract Wikipedia team (26Q2 (Oct–Dec)), 07Essential-Work: WF memcached service is dc-local but used for dc-global content - https://phabricator.wikimedia.org/T411807 (10Jdforrester-WMF) 03NEW p:05Triage→03Unbreak! [18:10:15] 06serviceops: Improve hcaptcha-proxy Grafana dashboard - https://phabricator.wikimedia.org/T411254#11433789 (10Raine) 05Open→03In progress p:05Triage→03Medium [19:10:48] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11434240 (10Scott_French) Summarizing discussion with @BTullis: The "toolbox" deployment exists to support ad-hoc manual tasks (e.g., via `kubectl exec`) and is rarely used in pract... [22:50:48] 06serviceops: Monitor hCaptcha (upstream) status - https://phabricator.wikimedia.org/T411255#11435016 (10Raine) 05Open→03Declined The monitoring created in T404204 is sufficient -- see https://grafana-rw.wikimedia.org/d/441b2def-52e9-49d6-acad-91f5bb748989/hcaptcha-reverse-proxy-proxoid?viewPanel=panel-3... [22:59:19] 06serviceops: hcaptcha-proxy: create logstash dashboard - https://phabricator.wikimedia.org/T411780#11435033 (10Raine) 05Open→03In progress p:05Triage→03Medium [22:59:49] 06serviceops, 07Documentation, 05WE4.2 Bot detection: hcaptcha proxy: update wikitech page - https://phabricator.wikimedia.org/T411131#11435037 (10Raine) [23:12:48] 06serviceops, 06SRE Observability, 06MediaWiki-Platform-Team (Kanban Board): MediaWiki periodic job startupregistrystats-mediawikiwiki failed - https://phabricator.wikimedia.org/T410764#11435052 (10matmarex) a:05matmarex→03None