[07:48:21] 06serviceops, 06Traffic, 10WE4.2 Bot detection (WE4.2 hCaptcha account creation trial): Investigate options for per-wiki, percentage-based rollout of hCaptcha - https://phabricator.wikimedia.org/T404184#11183709 (10kostajh) a:03kostajh There's some discussion [[ https://wikimedia.slack.com/archives/C01DFMX... [09:25:14] hey folks! [09:25:18] https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=alertname%3DKafka%20broker%20TLS%20certificate%20validity [09:25:38] kafka main's TLS certs are almost expired, the new ones should have already been generated [09:26:00] so it should just be a matter of doing a roll restart of main eqiad/codfw [09:26:34] I'll do kafka test [09:29:20] 06serviceops, 13Patch-For-Review: mysql.get_core_dbs() exception raised due to missing section - https://phabricator.wikimedia.org/T404464#11183972 (10elukey) New spicerack relase (0.11.7) done and rolled out to all the cumin nodes :) [14:36:43] 06serviceops, 06Content-Transform-Team, 13Patch-For-Review: Migrate parsoidtest1001 to PHP 8.3 - https://phabricator.wikimedia.org/T403772#11185381 (10Scott_French) [14:53:44] 06serviceops, 06Content-Transform-Team, 13Patch-For-Review: Migrate parsoidtest1001 to PHP 8.3 - https://phabricator.wikimedia.org/T403772#11185444 (10Scott_French) With some basic smoke testing after the upgrade to 8.3 (e.g., via [[ https://www.mediawiki.org/wiki/Parsoid/Round-trip_testing#Proxying_requests... [14:57:05] 06serviceops, 06Content-Transform-Team, 13Patch-For-Review: Migrate parsoidtest1001 to PHP 8.3 - https://phabricator.wikimedia.org/T403772#11185456 (10Scott_French) [15:32:17] 06serviceops, 13Patch-For-Review: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11185626 (10Scott_French) About 22 hours on, still no issues identified after migrating shellbox-media to 8.3. Given the that we're ~ 50% through shellbox instances (>> 90% of traffic)... [16:43:22] thanks for catching that, e.lukey! [16:45:19] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward DC Switchover (September 2025) - https://phabricator.wikimedia.org/T399894#11185996 (10EBlackorby-WMF) [16:47:47] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward DC Switchover (September 2025) - https://phabricator.wikimedia.org/T399894#11186006 (10EBlackorby-WMF) a:03EBlackorby-WMF [16:48:17] 06serviceops: mysql.get_core_dbs() exception raised due to missing section - https://phabricator.wikimedia.org/T404464#11186017 (10jasmine_) >>! In T404464#11183972, @elukey wrote: > New spicerack relase (0.11.7) done and rolled out to all the cumin nodes :) Thanks Luca! Major appreciation for releasing on such... [16:55:40] 06serviceops, 13Patch-For-Review: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#11186048 (10Dzahn) The `/var/lib/nginx` path is a separate mount point, using tmpfs, with a size of 4G. Apparently this is too small for known work loads as today there was an is... [17:02:31] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward DC Switchover (September 2025) - https://phabricator.wikimedia.org/T399894#11186056 (10EBlackorby-WMF) @jasmine_ would you be able to send an email to the wmfall mailing list? thank you! [17:06:25] 06serviceops: "kube-env: command not found" when in GNU screen - https://phabricator.wikimedia.org/T404739 (10Krinkle) 03NEW [17:06:42] 06serviceops: "kube-env: command not found" when in GNU screen - https://phabricator.wikimedia.org/T404739#11186076 (10Krinkle) [17:06:48] 06serviceops: "kube-env: command not found" when in GNU screen - https://phabricator.wikimedia.org/T404739#11186077 (10Krinkle) [17:24:18] 06serviceops, 06Infrastructure-Foundations: Increase RAM and nginx tmpfs on docker registry hosts - https://phabricator.wikimedia.org/T404742 (10Scott_French) 03NEW [17:26:12] 06serviceops, 06Infrastructure-Foundations: Increase RAM and nginx tmpfs on docker registry hosts - https://phabricator.wikimedia.org/T404742#11186196 (10CDanis) a:03CDanis [17:30:00] 06serviceops, 06Infrastructure-Foundations: Increase RAM and nginx tmpfs on docker registry hosts - https://phabricator.wikimedia.org/T404742#11186242 (10ops-monitoring-bot) VM registry1004.eqiad.wmnet rebooted by cdanis@cumin1003 with reason: None [17:36:48] 06serviceops, 06Infrastructure-Foundations: Increase RAM and nginx tmpfs on docker registry hosts - https://phabricator.wikimedia.org/T404742#11186280 (10ops-monitoring-bot) VM registry1005.eqiad.wmnet rebooted by cdanis@cumin1003 with reason: None [17:59:14] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review: Increase RAM and nginx tmpfs on docker registry hosts - https://phabricator.wikimedia.org/T404742#11186380 (10ops-monitoring-bot) VM registry1004.eqiad.wmnet rebooted by cdanis@cumin1003 with reason: None [18:03:56] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review: Increase RAM and nginx tmpfs on docker registry hosts - https://phabricator.wikimedia.org/T404742#11186384 (10ops-monitoring-bot) VM registry1005.eqiad.wmnet rebooted by cdanis@cumin1003 with reason: None [18:04:13] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review: Increase RAM and nginx tmpfs on docker registry hosts - https://phabricator.wikimedia.org/T404742#11186393 (10Scott_French) [18:09:35] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review: Increase RAM and nginx tmpfs on docker registry hosts - https://phabricator.wikimedia.org/T404742#11186425 (10ops-monitoring-bot) VM registry2004.codfw.wmnet rebooted by cdanis@cumin1003 with reason: None [18:14:46] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review: Increase RAM and nginx tmpfs on docker registry hosts - https://phabricator.wikimedia.org/T404742#11186452 (10ops-monitoring-bot) VM registry2005.codfw.wmnet rebooted by cdanis@cumin1003 with reason: None [18:46:11] 06serviceops, 06Infrastructure-Foundations: Increase RAM and nginx tmpfs on docker registry hosts - https://phabricator.wikimedia.org/T404742#11186558 (10CDanis) [18:58:41] 06serviceops, 06Infrastructure-Foundations: Increase RAM and nginx tmpfs on docker registry hosts - https://phabricator.wikimedia.org/T404742#11186606 (10CDanis) 05Open→03Resolved FWIW: I think the new considerations in the task description make it pretty reasonable to take ~1.1% of the total RAM alrea... [20:29:19] 06serviceops, 13Patch-For-Review: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11187124 (10Scott_French) Although the [[ https://sal.toolforge.org/log/5qWXU5kB8tZ8Ohr0FdzR | codfw ]] and [[ https://sal.toolforge.org/log/QpMcVJkBvg159pQrVfc8 | eqiad ]] updates got... [20:30:05] 06serviceops, 13Patch-For-Review: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11187125 (10Scott_French)