[00:09:51] 06Traffic: ncmonitor: Add "Depends-on" to Gerrit patches - https://phabricator.wikimedia.org/T401258#11067055 (10BCornwall) 05Open→03Declined Actually, that makes a pretty fatal assumption that each domain needs the same types of changes if approaching this from a simple "make the batch of patches ordere... [00:31:09] FIRING: LVSHighCPU: The host lvs1018:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs1018 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [00:36:09] RESOLVED: LVSHighCPU: The host lvs1018:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs1018 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [03:11:24] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Move pfw1b-codfw to rack F5 - https://phabricator.wikimedia.org/T401297#11067093 (10Papaul) Please see below for the migration diagram. {F65720751} [06:23:33] 06Traffic, 06collaboration-services, 10Gerrit, 10Release-Engineering-Team (Radar): Separate Gerrit https and ssh/git hostnames - https://phabricator.wikimedia.org/T394271#11067182 (10Jelto) a:05Jelto→03None I'll un-assign this task while I'm out but this task is still high-priority to ensure Gerrits av... [10:30:31] 10netops, 06Infrastructure-Foundations, 06SRE: Allow read-only users to view logs on Juniper devices - https://phabricator.wikimedia.org/T401378 (10cmooney) 03NEW p:05Triage→03Low [10:50:34] 06Traffic, 10Hiddenparma, 06SRE: Browser behaviour detection at the edge - https://phabricator.wikimedia.org/T400270#11067785 (10Vgutierrez) a:03Vgutierrez [11:22:33] 06Traffic, 10Hiddenparma, 06SRE: Browser behaviour detection at the edge - https://phabricator.wikimedia.org/T400270#11067835 (10Vgutierrez) https://phabricator.wikimedia.org/P80962 for future reference [11:35:42] 06Traffic, 06Data-Engineering: Reduce noise from duplicate sequence-gap alerts on HaProxy-webrequests - https://phabricator.wikimedia.org/T401383 (10Antoine_Quhen) 03NEW [11:57:00] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Move pfw1b-codfw to rack F5 - https://phabricator.wikimedia.org/T401297#11067948 (10cmooney) Thanks @papaul for the info. For the most part the diagram looks ok, a few questions/notes: The new switches should be called //fasw1-f5a// and //fasw... [13:16:37] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Move pfw1b-codfw to rack F5 - https://phabricator.wikimedia.org/T401297#11068239 (10Papaul) @cmooney please see answers and comments below. >>! In T401297#11067948, @cmooney wrote: > Thanks @papaul for the info. For the most part the diagram l... [14:04:19] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Move pfw1b-codfw to rack F5 - https://phabricator.wikimedia.org/T401297#11068404 (10cmooney) > "I think we have two cluster control / HA ports on each unit? > em0 and em1 as reported by the box, labelled HA0 and HA1 on the front? > We should u... [14:08:59] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11068426 (10elukey) Brain dump before I go on holidays, if anything is needed and I am not around. The current list of issues are: 1) The hosts are iDRAC 10, s... [14:09:38] sukhe: o/ I am going on holidays during the next couple of weeks so I summarized what I know about the new cp2xxx hosts here https://phabricator.wikimedia.org/T392851#11068425 [14:10:23] elukey: thanks a lot for all your work on this! have a restful break [14:10:32] we will see if we can pick up some of these bits [14:12:06] sukhe: Thanks! possibly the late_command.sh issue seems to be the most actionable one, and we'll have to solve it anyway [14:12:29] yep, thanks for the summary! [15:27:20] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Move pfw1b-codfw to rack F5 - https://phabricator.wikimedia.org/T401297#11068877 (10Papaul) I Still strongly disagree we need redundancy on the HA port. The reason being that if the port goes down, this will not have any impact. On the other ha... [15:53:05] 06Traffic, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 06SRE: Set expiry time for GeoIP cookies - https://phabricator.wikimedia.org/T122097#11068965 (10greg) Moving this out of unscheduled into Triage for us (FR Tech) to re-review/prioritize our side on it as it's a thing that needs cross-... [17:57:15] 06Traffic, 06DC-Ops, 10ops-esams, 10ops-magru, 13Patch-For-Review: CPU temperature issues in cp hosts - https://phabricator.wikimedia.org/T373993#11069441 (10RobH) >>! In T373993#11052738, @BCornwall wrote: > Re-assigning to @RobH: Rob, can you check the hot aisle in magru for us? I can, but can you adv... [18:01:08] 06Traffic, 06DC-Ops, 10ops-esams, 10ops-magru, 13Patch-For-Review: CPU temperature issues in cp hosts - https://phabricator.wikimedia.org/T373993#11069462 (10ssingh) >>! In T373993#11069441, @RobH wrote: >>>! In T373993#11052738, @BCornwall wrote: >> Re-assigning to @RobH: Rob, can you check the hot aisl... [18:02:07] 06Traffic, 06DC-Ops, 10ops-esams, 10ops-magru, 13Patch-For-Review: CPU temperature issues in cp hosts - https://phabricator.wikimedia.org/T373993#11069464 (10RobH) Ok, if Willy asked then I can put in the ticket no worries. I was asking so I could include it in the reasoning for the ticket later to him! [19:22:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2019 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [19:27:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2019 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [19:38:11] 06Traffic, 06SRE: Setting up Wikimedia Trust and Safety Help Center with Zendesk product: Seeking Guidance on host mapping - https://phabricator.wikimedia.org/T400952#11069639 (10Dzahn) a:05jhathaway→03None [19:51:33] 06Traffic, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 07fr-acoustic: Acoustic SMS: Domain needed for short links - https://phabricator.wikimedia.org/T379318#11069658 (10BCornwall) 05Open→03Resolved a:03BCornwall Resolving. Please re-open if this needs mo... [20:47:40] 06Traffic, 10DNS, 06SRE: Set mediawiki.gr, wikipedia.pt, and wiktionary.org.uk NS records to WMF - https://phabricator.wikimedia.org/T401438#11069798 (10Dzahn) re: wiktionary.org.uk - My first thought was that Wikimedia UK chapter might want this and/or we should redirect it to https://wikimedia.org.uk/ - Le... [20:49:46] 06Traffic, 10DNS, 06SRE: Set mediawiki.gr, wikipedia.pt, and wiktionary.org.uk NS records to WMF - https://phabricator.wikimedia.org/T401438#11069800 (10Dzahn) re: mediawiki.gr - similarly I would think let's ask https://wikimedia.gr Greek chapter if they know or have opinions. [22:03:55] 06Traffic, 10DNS, 06SRE: Set mediawiki.gr, wikipedia.pt, and wiktionary.org.uk NS records to WMF - https://phabricator.wikimedia.org/T401438#11069994 (10BCornwall) Emails sent to @Mike_Peel and @Geraki and brazenly subbed them here too :)