[04:48:46] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Marostegui) m2-master failed over from dbproxy1013 to dbproxy1015. Once the maintenance is done we need to revert this. [04:49:14] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Marostegui) [05:07:43] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Bstorm) [05:08:30] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Bstorm) [06:23:29] topranks: nice catch on https://phabricator.wikimedia.org/T287238#7237410 ! [06:35:39] Haha thanks. Just glad I was able to prove the network innocent! [07:56:00] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10MoritzMuehlenhoff) [08:26:30] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10elukey) [08:59:22] Actually John/Moritz, not sure if you've seen Luca's last comment on the above task XioNoX linked? [08:59:54] Seems he had trouble installing a more recent iptables-legacy package from buster-backports, which is required to fix the issue. [09:00:04] Just a heads up anyway if you weren't aware. [09:37:08] 10Mail, 10Infrastructure-Foundations: Upgrade MXes to Bullseye - https://phabricator.wikimedia.org/T286911 (10Majavah) The exim version in Bullseye (4.94) had some breaking changes - see https://www.debian.org/releases/bullseye/amd64/release-notes/ch-information.en.html#idm1404. So I agree that option 2 is a b... [09:37:47] topranks: ack, I left a comment on the task [12:08:47] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10cmooney) [12:13:40] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10cmooney) [12:14:50] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10cmooney) [12:38:48] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10MoritzMuehlenhoff) [12:39:05] 10Puppet, 10Infrastructure-Foundations, 10Patch-For-Review: puppetdb seems to be slow on host reimage - https://phabricator.wikimedia.org/T263578 (10jbond) So there was an issue yesterday when the autovacum process kicked in and caused a [[ https://grafana.wikimedia.org/d/000000469/postgres?viewPanel=1&orgId... [12:54:00] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10MoritzMuehlenhoff) [12:54:51] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10MoritzMuehlenhoff) [13:02:13] 10Puppet, 10Infrastructure-Foundations: puppetdb: tune postgress instance - https://phabricator.wikimedia.org/T287672 (10jbond) [13:03:00] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: puppetdb: tune postgress instance - https://phabricator.wikimedia.org/T287672 (10jbond) [13:03:12] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: puppetdb: tune postgress instance - https://phabricator.wikimedia.org/T287672 (10jbond) p:05Triage→03High [13:03:36] 10Puppet, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-jbond: puppetdb seems to be slow on host reimage - https://phabricator.wikimedia.org/T263578 (10jbond) [13:06:20] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: puppetdb Investigate the expected bahaviour of the faces table - https://phabricator.wikimedia.org/T287673 (10jbond) [13:06:33] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: puppetdb Investigate the expected bahaviour of the faces table - https://phabricator.wikimedia.org/T287673 (10jbond) p:05Triage→03Medium [13:08:38] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: puppetdb: filter large factsets - https://phabricator.wikimedia.org/T287674 (10jbond) [13:08:55] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: puppetdb: filter large factsets - https://phabricator.wikimedia.org/T287674 (10jbond) p:05Triage→03Medium [13:19:28] 10Puppet, 10Infrastructure-Foundations, 10PostgreSQL, 10User-jbond: puppetdb: tune postgress instance - https://phabricator.wikimedia.org/T287672 (10jbond) [13:29:03] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10MoritzMuehlenhoff) [13:39:09] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: puppetdb Investigate the expected bahaviour of the faces table - https://phabricator.wikimedia.org/T287673 (10jbond) I wonder if the increasing space is some how related to failing or suboptimal vacuuming and perhaps we should schedule a full vacum. from... [14:31:41] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team (Kanban): Cookbooks repository: avoid stale code in master branch - https://phabricator.wikimedia.org/T287465 (10jbond) >>! In T287465#7244334, @Bstorm wrote: > I should mention here that the original purpose of the wmcs prefix in the master br... [14:38:41] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10ops-monitoring-bot) Icinga downtime set by mmandere@cumin1001 for 1:00:00 4 host(s) and their services with reason: Eqiad row A maintenance ` cp[1075-1... [14:45:27] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10ops-monitoring-bot) Icinga downtime set by mmandere@cumin1001 for 1:00:00 1 host(s) and their services with reason: Eqiad row A maintenance ` dns1001.w... [14:47:36] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Vgutierrez) [14:48:56] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10ops-monitoring-bot) Icinga downtime set by mmandere@cumin1001 for 1:00:00 1 host(s) and their services with reason: Eqiad row A maintenance ` lvs1013.e... [14:49:31] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: puppetdb Investigate the expected bahaviour of the edges table - https://phabricator.wikimedia.org/T287673 (10jbond) [14:50:07] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Vgutierrez) [14:50:15] 10Puppet, 10Infrastructure-Foundations, 10observability, 10User-jbond: puppetdb Investigate the expected bahaviour of the edges table - https://phabricator.wikimedia.org/T287673 (10colewhite) [15:05:18] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Marostegui) [15:07:24] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Marostegui) >>! In T286032#7245427, @Marostegui wrote: > m2-master failed over from dbproxy1013 to dbproxy1015. Once the maintenance is done we need to... [15:27:41] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Adjust egress buffer allocations on ToR switches - https://phabricator.wikimedia.org/T284592 (10cmooney) [15:28:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Adjust egress buffer allocations on ToR switches - https://phabricator.wikimedia.org/T284592 (10cmooney) [16:31:08] 10Puppet, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-jbond: puppetdb seems to be slow on host reimage - https://phabricator.wikimedia.org/T263578 (10jbond) I should also note that currently we dont know when the next vacume process will occuer and it is quite possible that when it dose it wil... [18:25:03] 10Puppet, 10Infrastructure-Foundations, 10observability, 10User-jbond: puppetdb Investigate the expected bahaviour of the edges table - https://phabricator.wikimedia.org/T287673 (10jbond) i have done some digging on the on disk size vs the database size which i think shows how much data we could potentiall... [19:49:57] 10Puppet, 10Infrastructure-Foundations: Gendered pronouns in README - https://phabricator.wikimedia.org/T287705 (10mdipietro) [20:20:19] 10Puppet, 10Infrastructure-Foundations, 10Patch-For-Review, 10Voice & Tone: Gendered pronouns in README - https://phabricator.wikimedia.org/T287705 (10Mahir256) [20:43:09] 10Puppet, 10Infrastructure-Foundations, 10Patch-For-Review, 10Voice & Tone: Gendered pronouns in README - https://phabricator.wikimedia.org/T287705 (10mdipietro) 05Open→03Resolved [21:34:20] 10netops, 10DC-Ops, 10SRE, 10ops-codfw, 10Wikimedia-Incident: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10Papaul) Dear Juniper Networks Customer, Thank you for returning your defective product in relation to your recently created RMA. This notification confirms that Juniper...