[05:27:07] 10serviceops, 10DBA, 10Performance-Team: Update wgLBFactoryConf for x2 to register only the local primary - https://phabricator.wikimedia.org/T316482 (10Marostegui) p:05Triage→03High Raising this to high. @cdanis please confirm this wouldn't break anything. What I have done is, set `min_replicas: 0` on x... [05:50:46] 10serviceops, 10Diffusion, 10Gerrit, 10serviceops-collab, and 2 others: Gerrit replication to codfw (gerrit-replica.wikimedia.org) stopped working after Gerrit 3.4.5 upgrade - https://phabricator.wikimedia.org/T315942 (10hashar) 05Open→03Resolved I have updated both Gerrit to 3.4.5 and I have confirmed... [05:55:43] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install new codfw memcached hosts - https://phabricator.wikimedia.org/T313966 (10Joe) [05:55:53] 10serviceops, 10SRE: codfw (2) memcached host service implementation tracking - https://phabricator.wikimedia.org/T313968 (10Joe) 05Open→03In progress p:05Triage→03Medium [08:34:07] 10serviceops, 10Infrastructure-Foundations, 10netbox: Netbox and Redis - https://phabricator.wikimedia.org/T311385 (10ayounsi) Sounds like a great 1st step (if we want to test active active later on), if not final step (if we keep it as it). So I guess onthe Netbox side we "just" need those config options (... [08:46:34] 10serviceops, 10Infrastructure-Foundations, 10netbox: Netbox and Redis - https://phabricator.wikimedia.org/T311385 (10Joe) Silly question: do we have an idea of the size of the cached dataset? if it's small, do we need to keep redis remote to the VM where netbox runs, or should we install it as a local sidecar? [08:49:59] <_joe_> claime: and let's post the link here for other folks to see [08:52:55] https://etherpad.wikimedia.org/p/ServiceOps-AugustReboots [08:53:42] Keep in mind that's what I found through the cumin alias, so there may be more that I don't know about [10:14:33] 10serviceops, 10Patch-For-Review: Cleanup profile::docker::engine::version - https://phabricator.wikimedia.org/T316341 (10Clement_Goubert) We may run into issues with some cloud VMs : ` deployment-docker-citoid01.deployment-prep.eqiad1.wikimedia.cloud deployment-docker-cxserver01.deployment-prep.eqiad1.wikimed... [10:31:26] 10serviceops, 10Maps: Re-import full planet data into eqiad - https://phabricator.wikimedia.org/T314472 (10jijiki) a:03jijiki [11:09:30] 10serviceops, 10Infrastructure-Foundations, 10netbox: Netbox and Redis - https://phabricator.wikimedia.org/T311385 (10ayounsi) > do we have an idea of the size of the cached dataset? Good question! I guess this is small compared to https://grafana.wikimedia.org/d/000000174/redis?orgId=1&viewPanel=9 `name=ne... [11:12:17] 10serviceops, 10Patch-For-Review: Productionise mc20[38-55] - https://phabricator.wikimedia.org/T293012 (10jijiki) a:03jijiki [11:12:45] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10jijiki) a:03jijiki [11:35:25] 10serviceops, 10Patch-For-Review: Cleanup profile::docker::engine::version - https://phabricator.wikimedia.org/T316341 (10Clement_Goubert) Instances created as `bullseye` instances with the correct `docker::engine` config to replace these : ` deployment-docker-citoid02.deployment-prep.eqiad1.wikimedia.cloud de... [12:01:12] 10serviceops, 10Generated Data Platform, 10Image-Suggestions, 10SRE, and 3 others: New Service Request Generated Datasets: Image Suggestions Service - https://phabricator.wikimedia.org/T304891 (10WDoranWMF) [12:01:34] 10serviceops, 10Image-Suggestions, 10SRE: Setup Initial Image Suggestion Service CI and k8s params/stubs - https://phabricator.wikimedia.org/T305154 (10WDoranWMF) 05Open→03Resolved a:03WDoranWMF [12:01:41] 10serviceops, 10Generated Data Platform, 10Image-Suggestions, 10SRE, and 3 others: New Service Request Generated Datasets: Image Suggestions Service - https://phabricator.wikimedia.org/T304891 (10hnowlan) [12:02:02] 10serviceops, 10Image-Suggestions, 10SRE, 10Patch-For-Review: Blubber setup for Image Suggestions Service - https://phabricator.wikimedia.org/T305155 (10hnowlan) 05Open→03Resolved [12:02:23] 10serviceops, 10DBA, 10Performance-Team: Update wgLBFactoryConf for x2 to register only the local primary - https://phabricator.wikimedia.org/T316482 (10CDanis) @Marostegui That looks correct to me. I can write a dbctl patch today to do this automatically, so you can still manage replicas in dbctl, if that... [12:05:33] 10serviceops, 10DBA, 10Performance-Team: Update wgLBFactoryConf for x2 to register only the local primary - https://phabricator.wikimedia.org/T316482 (10Marostegui) >>! In T316482#8193428, @CDanis wrote: > @Marostegui That looks correct to me. > > I can write a dbctl patch today to do this automatically, so... [12:07:15] 10serviceops, 10DBA, 10Performance-Team: Update wgLBFactoryConf for x2 to register only the local primary - https://phabricator.wikimedia.org/T316482 (10CDanis) >>! In T316482#8193431, @Marostegui wrote: > omit-replicas looks good to me, so we can re-use it somewhere else if needed (ideally I would like to h... [12:16:05] 10serviceops, 10DBA, 10Performance-Team: Update wgLBFactoryConf for x2 to register only the local primary - https://phabricator.wikimedia.org/T316482 (10Marostegui) Sounds good, thank you @CDanis [12:30:34] 10serviceops, 10Dumps-Generation, 10Patch-For-Review, 10Performance-Team (Radar): Migrate WMF production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736 (10Jdforrester-WMF) [12:40:03] 10serviceops, 10SRE, 10decommission-hardware, 10ops-eqiad, 10Patch-For-Review: decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301) - https://phabricator.wikimedia.org/T280203 (10ayounsi) [13:38:32] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes102[34] - https://phabricator.wikimedia.org/T313873 (10akosiaris) [13:38:49] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes102[34] - https://phabricator.wikimedia.org/T313873 (10akosiaris) >>! In T313873#8189120, @Jclark-ctr wrote: > @akosiaris Can you verify host names? kubernetes102[01] Already in use Racking task T290202 Indeed. My mistake. Upd... [13:39:54] 10serviceops, 10Dumps-Generation, 10Patch-For-Review, 10Performance-Team (Radar): Migrate WMF production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736 (10Joe) >>! In T271736#8168864, @Krinkle wrote: >>>! In T271736#8160364, @Joe wrote: >> My current plan is to proceed as follows: >>... [14:10:06] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, and 2 others: Migrate thumbor to Kubernetes - https://phabricator.wikimedia.org/T233196 (10hnowlan) [14:30:40] 10serviceops, 10Data-Persistence (Consultation), 10MediaWiki-extensions-Phonos, 10SRE, 10Community-Tech (CommTech-Sprint-32): SRE/Data Persistence consultation — use of FSFileBackend for caching audio files - https://phabricator.wikimedia.org/T314789 (10JMcLeod_WMF) [14:32:18] 10serviceops, 10SRE, 10Wikimedia-Etherpad: Upgrade etherpad.wikimedia.org to (more) recent Etherpad version with more rich end-user features - https://phabricator.wikimedia.org/T316421 (10akosiaris) p:05Triage→03Low I am not sure I see what are the extra features either. Changelog (@JeanFred is correct r... [14:53:09] 10serviceops, 10serviceops-collab, 10GitLab (Infrastructure): Setup alerting for GitLab projects size limits - https://phabricator.wikimedia.org/T316553 (10Jelto) [15:05:02] 10serviceops, 10SRE, 10Wikimedia-Etherpad: Upgrade etherpad.wikimedia.org to (more) recent Etherpad version with more rich end-user features - https://phabricator.wikimedia.org/T316421 (10JeanFred) Sounds to me that this task should be split up: * renaming this one to “Minor upgrade of Etherpad from 1.8.16 t... [15:10:23] 10serviceops: Update the videoscaler alert to point at the correct runbook - https://phabricator.wikimedia.org/T316560 (10LSobanski) [15:17:10] 10serviceops: Update the videoscaler alert to point at the correct runbook - https://phabricator.wikimedia.org/T316560 (10RhinosF1) The way these docs work. You have to add a link to it as the run book is the same for all probe down alerts. [15:17:12] 10serviceops: Update the videoscaler alert to point at the correct runbook - https://phabricator.wikimedia.org/T316560 (10RhinosF1) [15:30:07] One thing I forgot to mention, could someone familiar with videoscalers take a look at https://phabricator.wikimedia.org/T316560 and check if the proposed documentation makes sense or if new one needs to be written? [15:52:09] 10serviceops, 10SRE, 10Wikimedia-Etherpad: Upgrade etherpad.wikimedia.org to (more) recent Etherpad version with more rich end-user features - https://phabricator.wikimedia.org/T316421 (10akosiaris) >>! In T316421#8194578, @JeanFred wrote: > Sounds to me that this task should be split up: > * renaming this o... [15:56:10] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review: Run stress tests on docker images infrastructure - https://phabricator.wikimedia.org/T264209 (10dancy) [15:56:24] 10serviceops, 10MW-on-K8s, 10Kubernetes, 10Patch-For-Review: Kubernetes timeing out before pulling the mediawiki-multiversion image - https://phabricator.wikimedia.org/T284628 (10dancy) 05Open→03Resolved a:03dancy [16:18:05] 10serviceops, 10Parsoid, 10Patch-For-Review, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10Clement_Goubert) `parse1001.eqiad.wmnet` pooled in place of `wtp1034.eqiad.wmnet` Now serving 4% of parsoid traffic from php7.4 only. [17:09:52] 10serviceops: Update the videoscaler alert to point at the correct runbook - https://phabricator.wikimedia.org/T316560 (10Dzahn) What RhinosF1 said. There is only one runbook URL per check and if everything uses the same "Probe Down" check then they will all have the same link. And combined with the move from I... [17:24:24] 10serviceops: Update the videoscaler alert to point at the correct runbook - https://phabricator.wikimedia.org/T316560 (10RLazarus) 05Open→03Resolved a:03RLazarus T312947 already tracks the larger question of how to organize runbooks for ProbeDown effectively. In the specific case, I agree with @LSobanski... [18:42:46] 10serviceops: Update the videoscaler alert to point at the correct runbook - https://phabricator.wikimedia.org/T316560 (10Dzahn) Oh, good point about using the link with an anchor and just creating that. Thanks. [19:09:17] 10serviceops, 10DBA, 10Performance-Team (Radar): Update wgLBFactoryConf for x2 to register only the local primary - https://phabricator.wikimedia.org/T316482 (10Krinkle) [20:17:07] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (Bonus Level 🕹️): Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10thcipriani) >>! In T313359#8095180, @Dzahn wrote: > Thank you very much to @RhinosF1 for https://www.... [20:17:54] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (Bonus Level 🕹️): Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10thcipriani) [20:38:50] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes102[34] - https://phabricator.wikimedia.org/T313873 (10Jclark-ctr) kubernetes1023 c6 u42 port 36 cableid 23000039 kubernetes1024 d8 u25 port 40 cableid 101760 [20:39:12] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes102[34] - https://phabricator.wikimedia.org/T313873 (10Jclark-ctr) [20:39:31] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes102[34] - https://phabricator.wikimedia.org/T313873 (10Jclark-ctr) a:05Jclark-ctr→03Cmjohnson [20:55:42] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (Bonus Level 🕹️): Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Dzahn) @thcipriani I think you can send it right now. It seems just about the value for $DATE in your...