[09:28:07] hi all i notice that mc2024 is failing to run puppet, it has now failed to do so for long enough that its been kicked out of puppetdb. seems the main reason is that its not in redis::shard (hieradata/common/redis.yaml) and so its hitting this block https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/redis/multidc.pp#L26-L28 [09:55:19] Morning all. I'd be grateful if someone could check my IPv6 and ASN allocation plan before I add them to netbox: https://phabricator.wikimedia.org/T310169#8156519 - Thanks. [10:34:29] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1001 for host gitlab2003.wikimedia.org with OS bullseye [10:50:39] <_joe_> claime: ^^ the cookbook also updates phabricator :D [10:55:01] _joe_: If given a -t, right ? [10:55:12] <_joe_> claime: yes [10:55:37] I'd seen it in the help, didn't put it in my notes, that's now done :p [11:09:22] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1001 for host gitlab2003.wikimedia.org with OS bullseye com... [11:13:19] claime, _joe_: we have 3 additional GitLab hosts (2 non production, 1 production) which need a reimage soon. Let me know if that could be interesting for you :) [11:13:44] <_joe_> jelto: get in line, I already have 24 hosts *at least* to reimage :P [11:15:52] _joe_: allright ;) 24 hosts sound like enough practise for the beginning [11:16:19] <_joe_> jelto: and in september, we'll have 400 or something [11:16:40] 👀 [12:22:16] 10serviceops, 10SRE, 10observability, 10Patch-For-Review, and 2 others: Create an alert for high memcached bw usage - https://phabricator.wikimedia.org/T224454 (10Aklapper) @CDanis: Only https://gerrit.wikimedia.org/r/c/operations/puppet/+/691216 is still open on this ticket, should that be merged or aband... [12:23:55] 10serviceops, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10SRE: schedule downtime for contint2001 - https://phabricator.wikimedia.org/T294271 (10hashar) [13:23:15] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar): Migrate WMF production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736 (10Joe) 05Open→03In progress a:03Joe [13:23:30] 10serviceops, 10MediaWiki-Releasing, 10PHP 7.2 support, 10PHP 7.3 support, 10Patch-For-Review: Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4 - https://phabricator.wikimedia.org/T261872 (10Joe) [13:41:36] 10serviceops, 10serviceops-collab: Review DNS TTLs for ServiceOps-Collab owned services - https://phabricator.wikimedia.org/T315319 (10LSobanski) [13:42:58] hi folks, as an heads-up I'm seeking a quick +1 for this (harmless) change https://gerrit.wikimedia.org/r/c/operations/puppet/+/822039 [13:43:23] I fell into that pitfall last week and thought we shouldn't be reading memcached.conf [14:49:58] no one? :( [15:05:00] I've not been to exposed to memcached - but if we don't use that file, the change sounds like a good idea [15:06:14] yeah AFAICT we do not use the file [15:06:59] i.e. the systemd unit has all configuration options [15:09:07] where is it coming from? doesn't look like it's in the memcached package? [15:09:19] should it just be ensure => absented? [15:09:32] good question. I was assuming it's in the package [15:10:11] mmhh I assumed it was in the package too, but didn't check! doing so now [15:10:57] yeah it does, from /usr/share/memcached/memcached.conf.default I think [15:11:39] I think we provisioned it via puppet until https://gerrit.wikimedia.org/r/c/operations/puppet/+/487898 [15:11:57] oh wait no, i'm misreading that patch [15:12:48] yeah I think that's right [15:13:07] if we absent the file the memcached package is going to put it back tho [15:13:44] yeah [15:13:49] your patch sgtm then [15:14:44] cheers both! appreciate it [21:17:28] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: kubernetes202[01] implementation tracking - https://phabricator.wikimedia.org/T313871 (10Papaul) @akosiaris we have already kubernetes202[01] so we have to use kubernetes202[34] Thanks [21:18:38] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes202[34] - https://phabricator.wikimedia.org/T313870 (10Papaul) [22:04:14] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Benchmark performance of MediaWiki on k8s - https://phabricator.wikimedia.org/T280497 (10aaron) [22:04:20] 10serviceops, 10Performance-Team: Investigate performance degradation at high concurrencies in php-fpm - https://phabricator.wikimedia.org/T293630 (10aaron) 05Open→03Resolved Tim, Timo, and I looked at the apcu graphs and do not see a need for fragmentation avoidance (e.g. via pruning) nor limiting space.... [22:40:19] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹ī¸): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) @thcipriani So far I am expecting to copy /srv/repos and /srv/dumps from old to new phab servers. with... [23:26:59] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹ī¸): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) Other things in /srv on phab1001 are: ` 871M /srv/deployment 1.6G /srv/dumps 4.0K /srv/git.wikimedia... [23:34:46] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹ī¸): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) regarding the UIDs.. user 'phd' has a reserved UID of 498. per docs (https://wikitech.wikimedia.org/wi...