[00:00:16] 10serviceops, 10SRE, 10Wikimedia-production-error: PHP7 corruption reports in 2020-2022 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10Krinkle) [07:52:38] hello folks [07:53:00] today at around 15UTC I'll work with dcops to upgrade nic+bios of kafka-main100[1-3] [07:53:14] one at the time, they will have to shutdown for some minutes [07:53:25] last step before the reimage :) [07:57:35] <_joe_> elukey: great work, thanks a lot! [08:02:31] <3 [08:22:01] 10serviceops, 10MW-on-K8s, 10SRE-swift-storage, 10Shellbox, 10Patch-For-Review: Support large files in Shellbox - https://phabricator.wikimedia.org/T292322 (10Joe) After the deployment, running the same command as before results in: ` var_dump($result); object(Shellbox\Command\BoxedResult)#675 (4) { ["... [10:53:12] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.1.1 - https://phabricator.wikimedia.org/T298986 (10Joe) p:05Triage→03Medium [11:10:37] 10serviceops, 10MW-on-K8s, 10SRE-swift-storage, 10Shellbox, 10Patch-For-Review: Support large files in Shellbox - https://phabricator.wikimedia.org/T292322 (10Joe) >>! In T292322#7603446, @tstarling wrote: > Is the procedure the one documented at https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments... [11:58:29] 10serviceops, 10MW-on-K8s, 10SRE-swift-storage, 10Shellbox, 10Patch-For-Review: Support large files in Shellbox - https://phabricator.wikimedia.org/T292322 (10Joe) I tried to give more resources to the shellbox container, but that didn't matter much - I guess the shellout we're running is single-threaded... [12:25:28] 10serviceops, 10Data-Engineering, 10Data-Engineering-Kanban, 10observability, 10Patch-For-Review: Move kafka-jumbo to a fixed uid/gid - https://phabricator.wikimedia.org/T296990 (10BTullis) a:05BTullis→03elukey @elukey has offered to carry out this work after all. [12:49:57] 10serviceops, 10Phabricator, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10LSobanski) Should be preceded by https://phabricator.wikimedia.org/T296022. [13:49:01] 10serviceops, 10Data-Engineering, 10observability, 10Patch-For-Review: Move kafka clusters to fixed uid/gid - https://phabricator.wikimedia.org/T296982 (10elukey) [13:49:13] 10serviceops, 10Data-Engineering, 10observability, 10Patch-For-Review: Move kafka clusters to fixed uid/gid - https://phabricator.wikimedia.org/T296982 (10elukey) [13:49:23] 10serviceops, 10Data-Engineering, 10Data-Engineering-Kanban, 10observability, 10Patch-For-Review: Move kafka-jumbo to a fixed uid/gid - https://phabricator.wikimedia.org/T296990 (10elukey) 05Stalled→03Resolved Cluster running with new gid/uid! [14:33:42] 10serviceops, 10MW-on-K8s, 10SRE-swift-storage, 10Shellbox, 10Patch-For-Review: Support large files in Shellbox - https://phabricator.wikimedia.org/T292322 (10Joe) p:05Triage→03High [15:09:10] Going to start the kafka main maintenance folks [15:16:46] <_joe_> elukey: are you starting the reimages? [15:17:34] _joe_ nono stopping one node at the time for nic/bios upgrades [15:17:44] it takes 10/15 mins for each node (more or less) [15:21:01] 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10Jelto) I removed `helm2` from `deploy1001` and `deploy2001` by merging https://gerrit.wikimedia.org/r/753026. I tested the removal before on WMCS and a temporary pontoon setup before (se... [15:21:34] 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10Jelto) [15:52:27] 10serviceops, 10MW-on-K8s, 10SRE-swift-storage, 10Shellbox, 10Patch-For-Review: Support large files in Shellbox - https://phabricator.wikimedia.org/T292322 (10Joe) a:03Joe I ran the command locally (I think!) on mwmaint1002, and it took a comparable time to what it took calling shellbox - apparently th... [16:41:28] 10serviceops, 10Patch-For-Review: Upgrade kafka-main nodes to buster - https://phabricator.wikimedia.org/T296641 (10Cmjohnson) [16:41:53] kafka main maintenance done [17:10:09] <_joe_> elukey: <3 [17:45:29] 10serviceops, 10Data-Engineering, 10Patch-For-Review: Move kafka clusters to fixed uid/gid - https://phabricator.wikimedia.org/T296982 (10herron) [19:24:03] 10serviceops, 10SRE: Clean up old Docker images on deneb - https://phabricator.wikimedia.org/T287222 (10Dzahn) ` [deneb:~] $ sudo systemctl status package_builder_Clean_up_build_directory.service ● package_builder_Clean_up_build_directory.service - Delete builds older the 2 weeks Loaded: loaded (/lib/syste... [20:33:28] 10serviceops, 10SRE Observability, 10Performance-Team (Radar): Enable mediawiki appserver metrics for jobrunner hosts - https://phabricator.wikimedia.org/T293943 (10lmata)