[07:00:45] 10serviceops, 10Performance-Team, 10SRE: WARNING: opcache cache-hit ratio is below 99.99% on multiple eqiad appservers and parsoid servers - https://phabricator.wikimedia.org/T287792 (10MoritzMuehlenhoff) p:05Triage→03Medium [07:02:56] 10serviceops, 10SRE: Clean up old Docker images on deneb - https://phabricator.wikimedia.org/T287222 (10MoritzMuehlenhoff) p:05High→03Medium [07:11:55] 10serviceops, 10DBA, 10User-fgiunchedi, 10cloud-services-team (Kanban): Roll restart haproxy to apply updated configuration - https://phabricator.wikimedia.org/T287574 (10dcaro) Ack, I'll be there :+1: [07:37:59] 10serviceops, 10DBA, 10User-fgiunchedi, 10cloud-services-team (Kanban): Roll restart haproxy to apply updated configuration - https://phabricator.wikimedia.org/T287574 (10fgiunchedi) Excellent, thank you @Andrew and @dcaro ! We're on for **Tues Aug 3rd at 9 UTC** [07:53:33] 10serviceops, 10SRE, 10Patch-For-Review: bring 43 new mediawiki appserver in eqiad into production - https://phabricator.wikimedia.org/T279309 (10Jelto) [08:08:36] 10serviceops, 10SRE: Clean up old Docker images on deneb - https://phabricator.wikimedia.org/T287222 (10ema) I've done some cleaning in my home too, down to ~500M now. [08:53:25] hello folks, if nobody opposes I'd drop the kubeyaml 1.12 support with https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/708725 [08:54:08] (kubeflow's yamls yield to a CI failure with 1.12) [08:56:16] elukey: go on [08:57:07] ack thanks :) [09:25:46] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10SRE, 10Patch-For-Review: Ensure the code is deployed to mediawiki on k8s when it is deployed to production - https://phabricator.wikimedia.org/T287570 (10Joe) The code should now be deployed when merged/built into an image within 5 minutes. I think... [10:56:08] hi! so I can decom 2 former mcrouter proxies without a replacement for the mcrouter proxy part, right? [11:42:31] 10serviceops, 10SRE, 10decommission-hardware, 10Patch-For-Review: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301) - https://phabricator.wikimedia.org/T280203 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mw1271.eqiad.wmnet` - m... [11:54:34] 10serviceops, 10SRE, 10decommission-hardware, 10Patch-For-Review: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301) - https://phabricator.wikimedia.org/T280203 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `mw1288.eqiad.wmnet` - m... [11:59:44] 10serviceops, 10SRE, 10decommission-hardware, 10Patch-For-Review: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301) - https://phabricator.wikimedia.org/T280203 (10Dzahn) [11:59:53] 10serviceops, 10SRE, 10decommission-hardware, 10Patch-For-Review: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301) - https://phabricator.wikimedia.org/T280203 (10Dzahn) a:03Dzahn [12:05:28] 10serviceops, 10SRE, 10decommission-hardware, 10Patch-For-Review: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301) - https://phabricator.wikimedia.org/T280203 (10Dzahn) 05Open→03Stalled everything is done except 4 canary API servers and decom'ing these is blocked by T273915 [13:54:31] 10serviceops, 10observability, 10GitLab (Initialization), 10Patch-For-Review: Define monitoring for gitlab - https://phabricator.wikimedia.org/T275170 (10Jelto) GitLab rails service has reduced availability roughly every ~24 hours (plus some offsets). See [this Grafana dashboard](https://grafana.wikimedia.... [15:05:10] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10SRE, and 2 others: Create a variant of mediawiki-multiversion which installs php-tideways-xhprof - https://phabricator.wikimedia.org/T287495 (10dancy) [16:03:07] 10serviceops, 10Performance-Team, 10SRE: WARNING: opcache cache-hit ratio is below 99.99% on multiple eqiad appservers and parsoid servers - https://phabricator.wikimedia.org/T287792 (10Legoktm) 05Open→03Resolved a:03Legoktm This appears to have resolved itself, if I had to guess this was caused by cod... [16:30:22] 10serviceops, 10observability, 10GitLab (Initialization), 10Patch-For-Review: Define monitoring for gitlab - https://phabricator.wikimedia.org/T275170 (10thcipriani) >>! In T275170#7251890, @Jelto wrote: > GitLab rails service has reduced availability roughly every ~24 hours (plus some offsets). See [this... [21:16:08] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10User-jijiki: (Need By: TBD) rack/setup/install mc10[37-54].eqiad.wmnet - https://phabricator.wikimedia.org/T274925 (10Jclark-ctr) @jijiki I am only held up on two servers for racking mc1039, mc1040 per racking proposal i am waiting for rack A7 to have 2... [21:48:05] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10User-jijiki: (Need By: TBD) rack/setup/install mc10[37-54].eqiad.wmnet - https://phabricator.wikimedia.org/T274925 (10Jclark-ctr) @jijiki If the remaining two work for being racked in A2 instead fo A7 they have been racked and can be configured by @Cmjohnso... [21:49:25] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10User-jijiki: (Need By: TBD) rack/setup/install mc10[37-54].eqiad.wmnet - https://phabricator.wikimedia.org/T274925 (10Jclark-ctr) a:05Jclark-ctr→03Cmjohnson [23:10:32] 10serviceops, 10SRE, 10decommission-hardware, 10Patch-For-Review: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301) - https://phabricator.wikimedia.org/T280203 (10wiki_willy) [23:38:53] 10serviceops, 10SRE, 10Services, 10Wikibase-Quality-Constraints, and 4 others: Deploy Shellbox instance (shellbox-constraints) for Wikidata constraint regexes - https://phabricator.wikimedia.org/T285104 (10Legoktm)