[06:41:39] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review, 10cloud-services-team (Kanban): Cookbooks repository: avoid stale code in master branch - https://phabricator.wikimedia.org/T287465 (10Volans) The above patches have all been merged and deployed. The add_wiki cookbook is now available as `sre.w... [07:35:17] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Modify homer/automation templates to support 100BaseTX interfaces with autoneg disabled. - https://phabricator.wikimedia.org/T288343 (10cmooney) p:05Triage→03Low [07:35:25] 10netops, 10Infrastructure-Foundations, 10SRE: Traffic Engineering for Anycast Ranges - https://phabricator.wikimedia.org/T288843 (10cmooney) p:05Triage→03Medium [07:38:29] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): Join ARIN waiting list to request additional IPv4 resources. - https://phabricator.wikimedia.org/T288342 (10cmooney) p:05Triage→03Low [07:38:41] 10netops, 10Infrastructure-Foundations, 10SRE: Create an alert for output discards on network devices - https://phabricator.wikimedia.org/T284593 (10cmooney) p:05Triage→03Medium [07:55:44] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: should we move $site global to a fact - https://phabricator.wikimedia.org/T289678 (10fgiunchedi) I like the idea of the datatype and having one list of sites we consider valid! I made the case on the review for why sticking with site is important, report... [10:44:02] 10Puppet, 10netops, 10Infrastructure-Foundations, 10SRE, and 2 others: LLDP: Ganeti hosts dont correctly report lldp_parent - https://phabricator.wikimedia.org/T289679 (10jbond) 05Resolved→03Open While rolling out the lldp factupdate i noticed an some machines have ip_forwarding enabled. this is likle... [11:19:23] hi, would it be too crazy to deploy a server access with no ssh access? I will of course ask for review before sending such a patch, but if curious the context is: https://phabricator.wikimedia.org/T289775#7314553 [11:20:01] summary is "web-access dependent on UNIX access" [11:21:25] I guess with a /bin/false shell or an invalid ssh key [11:39:27] 10Puppet, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE, and 2 others: Create a cron to clean clientbucket every day or hour - https://phabricator.wikimedia.org/T165885 (10Dzahn) >>! In T165885#7311068, @elukey wrote: > @jbond @Dzahn I got bitten by this problem in production 2/3 times as well (tod... [11:56:44] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Modify homer/automation templates to support 100BaseTX interfaces with autoneg disabled. - https://phabricator.wikimedia.org/T288343 (10cmooney) Sorted it eventually :) ` cmooney@mr1-ulsfo> show interfaces ge-0/0/0... [11:56:58] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Modify homer/automation templates to support 100BaseTX interfaces with autoneg disabled. - https://phabricator.wikimedia.org/T288343 (10cmooney) 05Open→03Resolved [12:08:52] jynus: analytics-privatedata-user with no ssh is common [12:08:56] And documented [12:09:02] ah, sorry [12:09:10] I didn't notice it [12:09:50] https://wikitech.wikimedia.org/wiki/Analytics/Data_access#Dashboards_in_web_tools_like_Turnilo_and/or_Superset_that_do_not_access_private_data [12:09:53] jynus: ^ [12:10:14] no, but that not's it [12:10:20] the user needs UNIX access [12:10:24] with no ssh [12:10:36] it NOT and LDAP-only thing [12:10:54] ah, I see what you mean, the one below [12:11:02] "This can be done by declaring the user in Puppet as usual, but with an empty array of ssh_keys" [12:11:07] cool then, thanks RhinosF1 [12:13:13] jynus: np [12:36:00] 10Puppet, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE, and 2 others: Create a cron to clean clientbucket every day or hour - https://phabricator.wikimedia.org/T165885 (10jbond) no issue with the change, however @elukey looking at an-launcher1002 there is 22GB of space free, if the filebucket is g... [12:53:08] 10Puppet, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE, and 2 others: Create a cron to clean clientbucket every day or hour - https://phabricator.wikimedia.org/T165885 (10jbond) I have fixed the issues on authdns and puppetmaster [12:57:42] 10Puppet, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE, and 2 others: Create a cron to clean clientbucket every day or hour - https://phabricator.wikimedia.org/T165885 (10jbond) @elukey next time you see the issue on an-launcher1002 can you run the two lines used above (for authdns and puppetmaste... [13:21:56] 10Puppet, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE, and 2 others: Create a cron to clean clientbucket every day or hour - https://phabricator.wikimedia.org/T165885 (10elukey) @jbond sure! [13:25:38] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team (Kanban): cloud cumin: exclude certain projects from "A:all" - https://phabricator.wikimedia.org/T289706 (10nskaggs) p:05Triage→03Low [13:27:03] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: should we move $site global to a fact - https://phabricator.wikimedia.org/T289678 (10jbond) Thanks @fgiunchedi i had net to go over thoses comments and update, and from the comments it was agreeaded to use site which im happy)ish) with. for posperity io... [13:35:12] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team (Kanban): Cookbooks repository: avoid stale code in master branch - https://phabricator.wikimedia.org/T287465 (10nskaggs) @Volans, as far as I can tell, I can no longer use this cookbook. The sudo rule no longer seems to work. [15:10:27] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE: (Need By: TBD) rack/setup/install atlas-codfw.wikimedia.org - https://phabricator.wikimedia.org/T273114 (10cmooney) Papaul rebooted yesterday with the USB key present. Things //appeared// to go ok, on the serial console the device went into a Linux... [15:41:33] 10netops, 10Infrastructure-Foundations, 10SRE: 2021-08-26 Primary inbound port utilisation over 80% page for mr1-esams.wikimedia.org - https://phabricator.wikimedia.org/T289820 (10jcrespo) Commenting as I think @ayounsi will not have been CCed on the original Phab report, for him to triage. [16:41:51] 10netops, 10Infrastructure-Foundations, 10SRE: 2021-08-26 Primary inbound port utilisation over 80% page for mr1-esams.wikimedia.org - https://phabricator.wikimedia.org/T289820 (10cmooney) I had a look at this this morning (didn't catch the page when it fired and it cleared quickly as you say). Seems to be... [16:42:11] 10netops, 10Infrastructure-Foundations, 10SRE: 2021-08-26 Primary inbound port utilisation over 80% page for mr1-esams.wikimedia.org - https://phabricator.wikimedia.org/T289820 (10cmooney) p:05Triage→03Low [18:06:55] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review, 10cloud-services-team (Kanban): Cookbooks repository: avoid stale code in master branch - https://phabricator.wikimedia.org/T287465 (10Volans) @nskaggs sorry for the trouble, there was a typo in the puppet patch, it should be fixed now. [18:59:45] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-eqiad, 10cloud-services-team (Hardware): (Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10RobH) [19:01:59] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-eqiad, 10cloud-services-team (Hardware): (Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10RobH) [19:03:48] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): CloudVPS: IPv6 early PoC - https://phabricator.wikimedia.org/T245495 (10faidon) [19:04:08] 10netops, 10Infrastructure-Foundations, 10SRE: Cloud IPv6 subnets - https://phabricator.wikimedia.org/T187929 (10faidon) 05Open→03Stalled There are some ongoing conversations with the WMCS team regarding the placement of their infrastructure in our network/infrastructure, and I think it would be good to... [19:05:27] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-eqiad, 10cloud-services-team (Hardware): Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10wiki_willy) [23:09:00] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10Spicerack, and 2 others: Clean up cron-specific elements of switchdc cookbooks - https://phabricator.wikimedia.org/T289078 (10Legoktm) 05Open→03Resolved I think this is all done now, woot!