[06:40:17] If I wanted to call some API endpoint once per minute and get an alert if the request fails, what's the best way to do that? An internal request through the service mesh would be fine. Do we have some service that does probes like that and pushed the result to prometheus? [09:32:20] 06serviceops, 06Content-Transform-Team, 06Traffic: Purging edge caches doesn't work for articles with ":" in their title - https://phabricator.wikimedia.org/T392849#10775677 (10Jgiannelos) FYI this is not reproduced on endpoints not migrated to rest-gateway yet. eg: * Given this page * https://en.wikipedia... [09:37:25] 06serviceops, 10MW-on-K8s, 06Trust and Safety Product Team, 10MediaModeration (MediaModeration 2.1), 13Patch-For-Review: Migrate MediaModeration jobs to mw-cron - https://phabricator.wikimedia.org/T385799#10775697 (10hnowlan) `mediamoderation-updateMetrics` appears to have run successfully this morning. [09:39:48] 06serviceops, 10MW-on-K8s, 06Trust and Safety Product Team, 10MediaModeration (MediaModeration 2.1), 13Patch-For-Review: Migrate MediaModeration jobs to mw-cron - https://phabricator.wikimedia.org/T385799#10775705 (10Dreamy_Jazz) I would agree with that. It appears to be working the same. [09:43:42] duesen: for existing services, this already happens, that's how k8s checks that the service is up [09:44:26] it checks /healthz by default, but it's configurable [09:45:02] Prometheus blackbox exporter is another option if you want more control than the k8s behaviour [09:45:17] What exactly are you trying to do? [09:57:36] 06serviceops, 06Content-Transform-Team, 06Traffic: Purging edge caches doesn't work for articles with ":" in their title - https://phabricator.wikimedia.org/T392849#10775748 (10hnowlan) From kafka - a successful enwiki purge and a failing testwiki purge: ` { "$schema": "/resource_change/1.0.0", "meta":... [10:22:33] 06serviceops, 06Content-Transform-Team, 06Traffic: Purging edge caches doesn't work for articles with ":" in their title - https://phabricator.wikimedia.org/T392849#10775807 (10Vgutierrez) from varnish point of view, after editing https://test.wikipedia.org/wiki/User:JGiannelos_(WMF)/test-pcs-rollout the fol... [10:27:51] 06serviceops, 06Content-Transform-Team, 06Traffic: Purging edge caches doesn't work for articles with ":" in their title - https://phabricator.wikimedia.org/T392849#10775829 (10Vgutierrez) a quick check shows that the URL receiving the PURGE is purged as expected: ` vgutierrez@carrot:~$ curl -4 'https://test... [10:49:01] 06serviceops, 06Content-Transform-Team, 06Traffic: Purging edge caches doesn't work for articles with ":" in their title - https://phabricator.wikimedia.org/T392849#10775884 (10Vgutierrez) ATS also shows how it's performing the request to the origin server after a PURGE: ` Date:2025-04-29 Time:10:40:51 ConnA... [11:57:12] 06serviceops, 10envoy, 10Observability-Metrics, 10SRE Observability (FY2024/2025-Q4): Revisit default envoy histogram buckets - https://phabricator.wikimedia.org/T391333#10776130 (10fgiunchedi) >>! In T391333#10744276, @akosiaris wrote: > Finally, how would we roll this out? We got multiple envoys right no... [13:20:30] 06serviceops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 13Patch-For-Review: Kubernetes dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390857#10776441 (10elukey) The changelog is very huge, but I think that V1beta1Eviction vs V1Eviction may be our only problem for the moment. I'... [13:39:12] 06serviceops, 06Infrastructure-Foundations: Redis dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390864#10776524 (10elukey) 05Open→03Resolved a:03elukey I checked the changelog for the functionality that we use and I think we are good if all the tests pass. I am curious though t... [15:10:59] 06serviceops, 10Deployments, 10Release-Engineering-Team (Radar), 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10776919 (10dancy) [17:38:15] 06serviceops: Remove PHP 7.4 from deployment hosts - https://phabricator.wikimedia.org/T392938 (10Scott_French) 03NEW [17:39:03] 06serviceops: Remove PHP 7.4 from deployment hosts - https://phabricator.wikimedia.org/T392938#10777773 (10Scott_French) p:05Triage→03Medium [17:39:24] 06serviceops: Remove PHP 7.4 from deployment hosts - https://phabricator.wikimedia.org/T392938#10777775 (10Scott_French) [17:39:33] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10777776 (10Scott_French) [19:19:10] 06serviceops, 06MediaWiki-Platform-Team: Migrate "startupregistrystats" maintenance script to k8s-mw-cron (mediawiki-platform-team) - https://phabricator.wikimedia.org/T388540#10778174 (10andrea.denisse) >>! In T388540#10774449, @Krinkle wrote: > While Prometheus counters are pretty straight-forward to agg... [20:35:59] 06serviceops, 06Release-Engineering-Team: train presync failed - https://phabricator.wikimedia.org/T387823#10778370 (10akosiaris) Change to allow #release-engineering-team members to start train-presync, train-clean and view logs has been merged and deployed. [21:19:45] 06serviceops, 10Observability-Metrics, 10SRE Observability (FY2024/2025-Q4): Repeated library panels in Grafana showing only after refresh, not on first load - https://phabricator.wikimedia.org/T384831#10778446 (10andrea.denisse) a:03andrea.denisse Hi @jijiki, thanks for the detailed bug report! We've upg... [21:52:41] 06serviceops, 10Observability-Metrics, 10SRE Observability (FY2024/2025-Q4): Repeated library panels in Grafana showing only after refresh, not on first load - https://phabricator.wikimedia.org/T384831#10778555 (10andrea.denisse) While testing, I noticed one of the panels shows an error about "query processi... [23:00:42] 06serviceops, 13Patch-For-Review: Build php-uuid package, and add to WMF production and CI - https://phabricator.wikimedia.org/T373752#10778629 (10Scott_French) Thanks, @Reedy - It's really not all that much effort to make this happen if it would help unblock you all. I have `php7.4-uuid` packages for `compon...