[00:17:20] <andrewbogott>	 taavi: here is a start at swift docs. I'm hoping you'll paste in some curl samples in the 'Using Swift and S3 APIs' section.
[00:17:31] <andrewbogott>	 (but not in the middle of the night of course)
[00:18:08] <andrewbogott>	 https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide
[07:31:25] <dcaro>	 morning
[07:34:26] <dcaro>	 there's something interesting happening with tools-sgeweblight-10-26, probably it's in a bad state, but when you try to run anything that needs a bit of memory it fails with no memory error ( failed to allocate memory: Cannot allocate memory), but free -m/htop etc show more than enough memory
[07:35:52] <dcaro>	 oh, now it does not happen :/
[07:36:22] <dcaro>	 puppet triggers it, but dpkg -l does not :/
[07:42:08] <dcaro>	 virsh also sees memory being available :/
[07:42:40] <dcaro>	 I'll reboot, but it's a weird behavior
[07:45:55] <dcaro>	 interesting...
[07:45:58] <dcaro>	 https://www.irccloud.com/pastebin/VuwRO6Hk/
[07:49:00] <taavi>	 i saw one memory error like that yesterday, also couldn't figure out why that was happening
[07:54:58] <dcaro>	 rebooting helped
[07:55:24] <dcaro>	 now I'm looking for the 503 above, and I'm seeing lots of '[None req-db0be433-d1a6-47e3-b3da-e1408bc73ddd novaadmin admin - - - -] volume service is down. (host: cloudcontrol1006@rbd)' (on logstash), looking
[07:56:59] <dcaro>	 hmm... it is down
[07:57:02] <dcaro>	 https://www.irccloud.com/pastebin/jn8W56lR/
[07:57:21] <dcaro>	 but no alerts on it :S
[07:57:44] <dcaro>	 looking
[07:59:07] <dcaro>	 hmm, puppet does not ensure running
[08:32:32] <dcaro>	 something is happening on the cloudsw
[08:32:39] <dcaro>	 summary: BGP CRITICAL - AS64605/IPv4: Active - Anycast
[08:32:44] <dcaro>	 topranks: ^
[08:33:57] <dcaro>	 it went away
[08:34:21] <dcaro>	 https://www.irccloud.com/pastebin/tKF0hkG6/
[08:36:14] <dcaro>	 I'll keep an eye, if anything else seems broken or it flaps again I'll open a task to follow up
[08:36:52] <dcaro>	 I'm going to do a page to test the metricsinfra setup, it should page only me but just fyi
[08:43:57] <dcaro>	 oh, the alerts on cloudsw are back
[08:44:39] <dcaro>	 hmm... and going away again, I'll open a task
[08:53:27] <dcaro>	 toolschecker started failing, looking
[08:53:55] <dcaro>	 opened T348839 for the switch <- topranks, XioNoX 
[08:53:56] <stashbot>	 T348839: [cloudsw1] BGP alert and port alert flapping - https://phabricator.wikimedia.org/T348839
[09:02:03] <topranks>	 dcaro: I updated the task - no issue on cloudsw really, it's a result of cloudservices2004-dev rebooting 
[09:02:07] <dcaro>	 the toolschecker alert is due to tools-sgeexec-10-8, it was running there, should be fixed now, alert should go away in a minute
[09:02:31] <dcaro>	 topranks: interesting
[09:02:44] <dcaro>	 topranks: thanks!
[09:03:21] <dcaro>	 anyone rebooted cloudservices2004-dev? x
[09:03:23] <dcaro>	 xd
[09:03:52] <dhinus>	 me!
[09:03:57] <dcaro>	 :)
[09:04:14] <dhinus>	 2005-dev as well
[09:04:19] <dcaro>	 good, maybe we might want to add a cookbook to silence alerts before-hand
[09:04:32] <dhinus>	 I used a cookbook :/
[09:05:14] <dcaro>	 then it might need extending, I think that by default sre downtime does not handle non-host specific alerts, that's one of the reasons why we wrote our own alertmanager handler (for ceph cluster stuff specifically)
[09:05:24] <dhinus>	 right
[09:05:31] <dcaro>	 so we might want to have a special treatment for cloudservices
[09:05:59] <dcaro>	 I always warted to have some kind of inventary of ost -> services running on it -> alerts for the service
[09:06:05] <dcaro>	 (dynamic if possible)
[09:08:06] <topranks>	 sudo cookbook sre.hosts.downtime --hours X "downtime switch before BGP peer reboot" -t XXXX --force "cloudsw1-b1-codfw, cloudsw1-b1-codfw IPv6, cloudsw1-b1-codfw.mgmt"
[09:08:31] <topranks>	 the downtime cookbook doesn't have a way right now to tell if a host is BGP peering with another device and then downtime that other device too 
[09:08:55] <topranks>	 something like the above would do it if required 
[09:10:16] <topranks>	 actually correction - you only need to downtime hostname "cloudsw1-b1-codfw.mgmt"
[09:10:29] <dcaro>	 I was wondering about the others xd
[09:11:18] <dhinus>	 I did restart that same host last week though, did it also trigger an alert?
[09:11:21] <dcaro>	 so you can downtime anything with 'sre.hosts.downtime'? I though it only set the 'instance=%s' label
[09:12:16] <dcaro>	 dhinus: maybe https://logstash.wikimedia.org/goto/2c8f9e8df11e6fcf05c83cb37d7cde32 ,  there's more triggers there
[09:12:28] <topranks>	 Yeah, I think the "--force" is needed for things not in puppetdb, not 100% but the --force is needed for network kit anyway 
[09:14:37] <dcaro>	 ack
[09:18:28] <dhinus>	 how many hosts have this kind of BGP peering? is it something that was introduced with the cloudlb project?
[09:18:35] <dcaro>	 dhinus: created T348841 for the cookbook, in case you feel like playing with it (no pressure)
[09:18:36] <dcaro>	 T348841
[09:18:36] <stashbot>	 T348841: [wmcs-cookbooks] add a cookbook to reboot a cloudservices host - https://phabricator.wikimedia.org/T348841
[09:18:42] <dcaro>	 dhinus: I think any cloudservices VM
[09:19:02] <dcaro>	 cloudlb should also no?
[09:19:35] <dcaro>	 https://www.irccloud.com/pastebin/7ii1D6u3/
[09:30:01] <dhinus>	 hmm some network tests are failing in codfw
[09:30:39] <dhinus>	 "failed test: VM (no floating IP) can contact recursor DNS server"
[09:30:47] <dhinus>	 and 3 other ones
[09:31:20] <dhinus>	 https://phabricator.wikimedia.org/P52926
[09:36:07] <dcaro>	 looking
[09:36:08] <topranks>	 resolver seems ok: https://phabricator.wikimedia.org/P52926#213996
[09:38:09] <dcaro>	 ssh: connect to host bastion.bastioninfra-codfw1dev.codfw1dev.wmcloud.org port 22: No route to host
[09:38:16] <dcaro>	 when trying to ssh to proxy-02.proxy-codfw1dev.codfw1dev.wikimedia.cloud
[09:38:33] <topranks>	 dhinus: cloudlb and cloudservices only ones right now 
[09:38:44] <dcaro>	 https://www.irccloud.com/pastebin/yrqbAzk7/
[09:38:56] <topranks>	 anything with the bird anycast profile in puppet 
[09:39:32] <topranks>	 dcaro: what is the source IP of that ping?
[09:39:44] <dcaro>	 topranks: my laptop :)
[09:39:53] <topranks>	 ok
[09:40:36] <topranks>	 cloudnet vip seems unreachable 
[09:42:32] <topranks>	 something is weird with cloudnet2005-dev / neutron 
[09:42:38] <topranks>	 root@cloudnet2005-dev:~# arping -I qg-1290224c-b1 185.15.57.11
[09:42:38] <topranks>	 arping: connect: Permission denied
[09:42:52] <dcaro>	 oh, that's weird
[09:43:03] <topranks>	 root@cloudnet2005-dev:~# ip -br addr show dev qg-1290224c-b1
[09:43:03] <topranks>	 qg-1290224c-b1@if17 UP             185.15.57.10/30 185.15.57.2/32 185.15.57.20/32 185.15.57.21/32 185.15.57.4/32 185.15.57.5/32 185.15.57.6/32 fe80::f816:3eff:fe35:9f97/64 
[09:43:03] <topranks>	 root@cloudnet2005-dev:~# 
[09:43:03] <topranks>	 root@cloudnet2005-dev:~# ping 185.15.57.11
[09:43:03] <topranks>	 ping: Do you want to ping broadcast? Then -b. If not, check your local firewall rules
[09:44:23] <topranks>	 the issue with the routing is cos cloudgw can't ARP for the next-hop, 185.15.57.11, which is the neutron VIP 
[09:44:54] <dcaro>	 https://www.irccloud.com/pastebin/KYLN2bY4/
[09:45:00] <dcaro>	 that's it's own ip
[09:45:03] <dcaro>	 no?
[09:45:12] <dcaro>	 (vlan2107@eno1)
[09:49:59] <topranks>	 dcaro: sorry was looking at something else, it can't arp for .10, typo on my side 
[09:50:39] <topranks>	 which is where is routes the networks neutron uses:
[09:50:43] <topranks>	 https://www.irccloud.com/pastebin/W92o40Qb/
[09:52:39] <dcaro>	 ack
[09:53:14] <topranks>	 it's arping out and getting no answer
[09:53:17] <topranks>	 https://www.irccloud.com/pastebin/yi1vpNjb/
[09:54:38] <dcaro>	 ok, so cloudgw seems not to get arp responses from 185.15.57.10 (the vrf)
[09:55:08] <dcaro>	 right?
[09:56:04] <topranks>	 yeah, speciffically cloudnet2005-dev doesn't respond 
[09:56:08] <topranks>	 https://www.irccloud.com/pastebin/3n9RoOVg/
[09:56:26] <topranks>	 presumably due to whatever weird kernel network buggyness doesn't let root do arping or ping on that interface 
[09:56:48] <dcaro>	 that sounds nasty :/ (the permission error)
[09:57:27] <dcaro>	 let me try to debug that
[09:58:00] <dcaro>	 I get a different error now
[09:58:02] <dcaro>	 https://www.irccloud.com/pastebin/yD0gwiPw/
[09:58:28] <topranks>	 you need a shell in the qrouter/neutron l3 agent namespace 
[09:58:29] <topranks>	 sudo ip netns exec qrouter-5712e22e-134a-40d3-a75a-1c9b441717ad bash
[09:58:37] <dcaro>	 ack
[09:58:39] <topranks>	 "sudo ip netns list" to get that name btw 
[09:58:56] <dcaro>	 ohhh... so that command is from inside a namespace... that might be something
[09:59:34] <topranks>	 it normally works just fine I've had to do it a few times to verify the cloudnet<->cloudgw comms 
[10:02:22] <dcaro>	 it's specifically the connect syscall
[10:02:22] <dcaro>	 connect(4, {sa_family=AF_INET, sin_port=htons(1025), sin_addr=inet_addr("185.15.57.11")}, 16) = -1 EACCES (Permission denied)
[10:02:37] <dcaro>	 (I was hoping it was trying to play with /proc xd)
[10:07:09] <topranks>	 actuaally that may be a misnomer, .11 is the broadcast on it's subnet
[10:07:17] <topranks>	 dcaro: apologies perhaps I led us the wrong way here 
[10:09:08] <dcaro>	 that's why ping complains with the broadcast xd
[10:09:24] <dcaro>	 why is then the cloudgw trying to arp it/
[10:09:26] <dcaro>	 ?
[10:10:14] <topranks>	 Only the VIP is on that range
[10:10:17] <topranks>	 https://www.irccloud.com/pastebin/dS75s8Xw/
[10:10:35] <topranks>	 I'm trying to dig out the task - we should change the neutron subnet so we don't get this
[10:10:59] <topranks>	 I think potentially there is an issue the cloudgw ARP table is only populating if the ARP is generated from cloudnet side 
[10:11:12] <topranks>	 if cloudgw asks the other side it uses the .11 IP, and cloudnet ignores 
[10:12:25] <topranks>	 https://phabricator.wikimedia.org/T348140#9236083
[10:13:20] <topranks>	 The netmask on this should be changed to /29:
[10:13:23] <topranks>	 https://www.irccloud.com/pastebin/a7qFfb6V/
[10:14:04] <topranks>	 everything else stays the same but cidr should be 185.15.57.8/29 
[10:15:17] <topranks>	 This is karma for me as I meant to follow up with Arturo earlier in the week on it and got distracted by other things 
[10:16:09] <taavi>	 can I help with anything?
[10:16:20] <taavi>	 (sorry I haven't really been following the conversation)
[10:16:31] <dcaro>	 okok, let me see if I'm understanding, so the cloudgw has ip 185.15.57.11, but on cloudnet side 185.15.57.11 is the broadcast for the 185.15.57.8/30 network (configured through openstack neutron)
[10:16:51] <dcaro>	 so it just ignores any arp request from that ip, so cloudgw does not get any replies?
[10:17:53] <topranks>	 taavi: do you know how to adjust a neutron subnet?
[10:18:39] <topranks>	 last week the cloudnet/neutron side of the cloudgw <-> cloudnet link netmask wasn't changed when the cloudgw was done 
[10:19:31] <taavi>	 you want to change the CIDR range of a subnet object?
[10:19:37] <topranks>	 yes 
[10:19:58] <topranks>	 the issue now is the cloudgw ARP for the neutron VIP only populates when cloudnet ARPs out to cloudgw 
[10:20:06] <topranks>	 if cloudgw tries to arp for cloudnet it fails 
[10:20:30] <taavi>	 `sudo wmcs-openstack subnet set --help` doesn't show an option for that, so it might need to be deleted and recreated
[10:21:21] <dcaro>	 I was looking into that too
[10:21:50] <dcaro>	 I think its
[10:21:52] <dcaro>	 https://www.irccloud.com/pastebin/ZFAKi8nF/
[10:22:20] <topranks>	 I'm not sure, currently it shows "none" for segment ID 
[10:22:41] <taavi>	 network segments are a separate concept: https://docs.openstack.org/neutron/latest/admin/config-network-segment-ranges.html
[10:22:48] <topranks>	 yeah
[10:23:05] <dcaro>	 yep
[10:23:16] <topranks>	 not sure if relevant: https://platform9.com/kb/openstack/increase-network-subnet-cidr-to-add-more-ip-addresses-in-subnet
[10:24:41] <topranks>	 redhat docs don't seem to show it as option under "neutron subnet-update" either, I suspect you may be right taavi 
[10:26:04] <dcaro>	 from the create options, it would be this I think:
[10:26:04] <dcaro>	 --subnet-pool <subnet-pool>
[10:26:04] <dcaro>	                         Subnet pool from which this subnet will obtain a CIDR (Name or ID)
[10:27:27] <taavi>	 there's a neutron port in the current subnet that we need to deal with: https://phabricator.wikimedia.org/P52927
[10:27:58] <taavi>	 that's connected to the cloudinstances2b-gw router
[10:28:22] <topranks>	 yeah that corresponds to the virtual nic on cloudnet2005-dev
[10:28:24] <dcaro>	 on the subnet side, it means that we are using all the ips until .24 (with the three subnets here):
[10:28:25] <topranks>	 https://www.irccloud.com/pastebin/pmkTUpgN/
[10:28:27] <dcaro>	 https://www.irccloud.com/pastebin/sJr6aL5y/
[10:28:45] <dcaro>	 and we will not be able to grow those anymore without moving the ranges
[10:29:44] <dcaro>	 but that's ok for now I guess? (no growth is planned on those ranges right?)
[10:29:46] <topranks>	 not sure about growing/expansion, we can deal with that later I think 
[10:29:56] <dcaro>	 yep
[10:30:00] <topranks>	 We just need to change the netmask for cloud-gw-transport-codfw for current issue 
[10:30:43] <topranks>	 dcaro: what command produced the last output you pasted?
[10:30:55] <dcaro>	 openstack subnet list
[10:31:06] <dcaro>	 (`wmcs-openstack subnet list` if you use the script)
[10:31:21] <topranks>	 cool yep thanks :)
[10:31:51] <taavi>	 (I have `alias os="sudo wmcs-openstack"` in my bashrc which makes commands much easier to type :P)
[10:32:31] <topranks>	 another part of the puzzle here - that would potentially solve this 
[10:32:36] <topranks>	 without touching neutron 
[10:32:50] <topranks>	 is to fix the /32 subnet mask on the cloudgw keepalived vip 
[10:33:29] <topranks>	 In /etc/keepalived/keepalived.conf 
[10:33:58] <topranks>	 Change this:
[10:34:01] <topranks>	 https://www.irccloud.com/pastebin/zy9u7Tg2/
[10:34:16] <topranks>	 to this:
[10:34:22] <topranks>	 https://www.irccloud.com/pastebin/CRotmdQH/
[10:34:50] <topranks>	 this would cause the cloudgw to ARP from source 185.15.57.9 I believe, which currently it won't do cos of the /32 mask 
[10:35:17] <dcaro>	 Feels kind of hacky to leave the subnet unmatched xd
[10:36:02] <topranks>	 sure it needs to change 
[10:36:14] <topranks>	 the /32 is hacky and needs to also change though 
[10:36:23] <dcaro>	 ack
[10:36:33] <topranks>	 we should fix whichever is easiest first to repair the problem, but do both ultimately 
[10:36:52] <dcaro>	 would it be easier to change later?
[10:46:28] <dcaro>	 the puppet code for the keepalived states that it's using /32 because of https://phabricator.wikimedia.org/T295774
[10:47:38] <dcaro>	 it seems our automation + netbox possibly? might not like them to be wider than /32 there
[10:48:04] <topranks>	 that's a mis-understanding - I raised it at Infra Foundations meeting and we've a plan to deal 
[10:48:16] <topranks>	 the driver for volans remakrs was admin / netbox side and to do with us importing VIPs
[10:48:26] <topranks>	 99% of which are LVS VIPs on lo interface, and should be /32
[10:48:48] <topranks>	 the assumption VIPs should be /32 in netbox doesn't match all scenarios, i.e. keepalived  / vrrp type VIPs
[10:48:56] <dcaro>	 agree
[10:49:44] <topranks>	 changing to /32 shouldn't really have been done - as on the wire it means we can get issues 
[10:50:03] <topranks>	 there were other tricks to work around this - but we removed them in T348140
[10:50:04] <stashbot>	 T348140: Change cloud-instance-transport vlan subnets from /30 to /29 - https://phabricator.wikimedia.org/T348140
[10:50:39] <topranks>	 we should have updated the netron netmask at same time I'm seeing now though, the inconsistency I thought would be ok until we could, but I didn't anticipate this edge case 
[10:50:44] <topranks>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/965708
[10:52:03] <dcaro>	 this also affects eqiad right?
[10:52:03] <dcaro>	 | 77dba34f-c8f2-4706-a0b6-2a8ed4d91f51 | cloud-gw-transport-eqiad            | 5c9ee953-3a19-4e84-be0f-069b5da75123 | 185.15.56.236/30 |
[10:52:33] <topranks>	 yep: https://phabricator.wikimedia.org/T348140#9228813
[10:53:44] <dcaro>	 hmmm... I think I'll grab a bite before playing with codfw, as things might break
[10:53:59] <dcaro>	 (break more)
[10:54:23] <topranks>	 I'm fairly sure the patch is safe and will fix the current issue 
[10:54:38] <topranks>	 perhaps taavi is brave enough to +1 :P
[10:54:53] <dcaro>	 oh, missed the patch
[10:55:10] <topranks>	 np, sry I'm spitting messages out rapid fire here :D
[10:55:27] <topranks>	 patch for cloudgw side, neutron change seems trickier to do so I agree let's not try to rush it 
[10:55:30] <taavi>	 not an eqiad one on a friday :D
[10:55:31] <dcaro>	 that patch is only for eqiad, can be do codfw first?
[10:55:59] <topranks>	 it's for the cloudgw role so it'd change both, I guess we can disable puppet on eqiad cloudgw's and make sure codfw is ok first?
[10:56:20] <dcaro>	 the hiera file is hieradata/role/eqiad/wmcs/cloudgw.yaml
[10:56:25] <taavi>	 the patch is changing the eqiad-specific file
[10:56:28] <dcaro>	 that should only affect eqiad no?
[10:56:35] <topranks>	 oh sorry my bad 
[10:56:46] <dcaro>	 xd, we are saying the same thing at the same time
[10:56:47] <topranks>	 ok scrap that - I'll leave it there let's do one for codfw 
[10:59:36] <topranks>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/965712
[11:02:54] <dcaro>	 can we pcc it? or feel free to merge if you are sure
[11:03:01] <topranks>	 dcaro: thanks for the +1, let me know if you're happy to merge or if we should hold off 
[11:03:29] <topranks>	 I think I'll merge, the change to the template is clear enough I think
[11:03:31] <topranks>	 thanks
[11:04:14] <dcaro>	 ack, let me know if I can help, otherwise I'm pinging around xd
[11:04:32] <topranks>	 ok, forcing puppet run on cloudgw2002-dev now 
[11:05:16] <dcaro>	 I see the change now in the config
[11:05:30] <dcaro>	 this works
[11:05:32] <topranks>	 keepalived flipped to other cloudgw after daemon restarted 
[11:05:33] <dcaro>	 https://www.irccloud.com/pastebin/hyLTkQPE/
[11:05:56] <topranks>	 yeah, although was sporadic before
[11:06:02] <topranks>	 these are stats I had over last ~30 mins
[11:06:03] <topranks>	 1523 packets transmitted, 1181 received, +263 errors, 22.4557% packet loss, time 1554009ms
[11:06:52] <topranks>	 ARP entry times out on cloudgw, it can't then arp for cloudnet 
[11:07:08] <topranks>	 stuff is broken until cloudnet sends an arp for cloudgw, which allows it to re-populate the table 
[11:07:22] <dcaro>	 hmm, this works now though
[11:07:24] <dcaro>	 https://www.irccloud.com/pastebin/Zqyouh7b/
[11:07:24] <topranks>	 So hopefully this will fix that 
[11:07:35] <topranks>	 yeah it's not been broken full time 
[11:08:02] <dcaro>	 ack, anything we can test to make sure it's working?
[11:08:04] <topranks>	 any yep I think it is fixed - but just saying it was working then not constantly last while 
[11:08:19] <dcaro>	 we can keep an eye for sure xd
[11:08:25] <topranks>	 I'll leave a ping running for next while and ensure we've 0% loss 
[11:08:58] <topranks>	 nope :(
[11:09:12] <topranks>	 https://www.irccloud.com/pastebin/gHHi0bRC/
[11:09:23] <dcaro>	 oh, so not fixed then?
[11:09:23] <topranks>	 ^^ although this may just be due to keepalived flip back 
[11:09:28] <topranks>	 let's see how it goes 
[11:09:31] <dcaro>	 okok
[11:10:15] <dcaro>	 I'll rerun the network tests a few times too
[11:11:07] <dcaro>	 they passed this time at least, that's a good signal cc.dhinus :
[11:11:09] <dcaro>	 https://www.irccloud.com/pastebin/3Da2TPFz/
[11:11:16] <topranks>	 cloudgw is sending arp's from the VIP now 
[11:11:19] <topranks>	 11:10:57.364805 2c:ea:7f:7b:e1:04 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 185.15.57.10 tell 185.15.57.9, length 28
[11:11:32] <topranks>	 https://www.irccloud.com/pastebin/FzqrgcSN/
[11:12:13] <dcaro>	 nice, I think that's proof enough that it's working now
[11:12:32] <dhinus>	 nice one, thanks all!
[11:13:33] <topranks>	 might need another change on cloudgw side, work-around until neutron subnet is updated 
[11:16:03] <dcaro>	 brb
[11:35:26] <topranks>	 folks patch here:
[11:35:27] <topranks>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/965720
[11:35:50] <topranks>	 I made the change manually in codfw to test first and it does the expected
[11:35:50] <topranks>	 https://phabricator.wikimedia.org/P52929
[11:36:42] <topranks>	 we shouldn't merge above patch before https://gerrit.wikimedia.org/r/c/operations/puppet/+/965708
[11:37:09] <topranks>	 to be safe - although I don't think when puppet runs it actually will adjust the routes as they only run at ifup 
[11:38:03] <topranks>	 actually - scrap that it won't take that 'src' on the backup node 
[11:49:57] <dcaro>	 so the first patch does not work?
[11:55:14] <topranks>	 The patch allows the cloudgw to send an ARP from the VIP - previously it couldn't 
[11:55:19] <topranks>	 so this now works following change:
[11:55:22] <topranks>	 https://www.irccloud.com/pastebin/NfhhCaHN/
[11:56:06] <topranks>	 The problem is in the above I've forced the source to be the VIP, so the ARP works 
[11:56:34] <topranks>	 codfw right now is fine, I manually modified routes on cloudgw2002-dev so it will do this when forwarding traffic for cloud ranges:
[11:57:02] <topranks>	 https://www.irccloud.com/pastebin/RdMYXDQH/
[11:57:19] <topranks>	 but that can't be replicated on the backup (VIP doesn't live there right now so won't accept the 'src')
[11:57:39] <topranks>	 so if things fail over we could have some packet loss again
[11:57:56] <topranks>	 ultimately we need to change the neutron subnet, no further change on cloudgw in puppet will fix it I think 
[12:29:05] <dcaro>	 ack, that means recreating the subnet in openstack, and that includes having to remove and re-add the cloudnet port (.10) right?
[12:29:33] <dcaro>	 I'm guessing that the removal will mean some downtime
[12:30:46] <topranks>	 yeah I think so, something perhaps like this script 
[12:30:47] <topranks>	 https://gist.github.com/jimleitch01/cf844a5d78f97073539d
[12:31:02] <topranks>	 but I've no experience with it tbh 
[12:31:41] <topranks>	 we shouldn't need to do all those bits - like touch the router or network, just remove what's needed to delete the subnet then re-add 
[12:31:51] <topranks>	 which I *think* should just be the port and subnet 
[12:39:22] <topranks>	 are there any docs arturo might have left us on how to set it up from scratch?
[12:39:24] <dcaro>	 oh, I think that there's more stuff, the router for example
[12:39:37] <dcaro>	 references that network
[12:39:38] <dcaro>	 | external_gateway_info   | {"network_id": "57017d7c-3817-429a-8aa3-b028de82cdcc", "external_fixed_ips": [{"subnet_id": "2596edb4-5a40-41b9-9e67-f1f9e40e329c", "ip_address": "185.15.57.10"}], "enable_snat": false}
[12:40:08] <topranks>	 that makes sense - I guess it's more of a parent/child ting 
[12:40:18] <dcaro>	 yep
[12:40:19] <topranks>	 i.e. can we delete the subnet, redefine it, and reconnect it to the router 
[12:40:28] <topranks>	 or do we need to delete the router also etc.
[12:40:37] <dcaro>	 I hope the first, let me check
[12:41:01] <dcaro>	 there's an 'add subnet' option for the router
[12:41:15] <dcaro>	 https://www.irccloud.com/pastebin/g0F3A1Mj/
[12:41:33] <dcaro>	 for 'set', I think it can be changed without recreating it
[12:41:35] <topranks>	 yeah at worst case we should probably remove subnet from router, re-create, then add subnet back to router 
[12:41:41] <dcaro>	 ack
[12:41:45] <topranks>	 but not actually delete the router
[12:50:30] <dcaro>	 topranks: I added some notes https://phabricator.wikimedia.org/T348140, can you fix/add what I missed? xd
[12:52:18] <topranks>	 looks good to me.  I am slightly nervous, I think our logic is correct but not having done it before I’m worried maybe if we miss a command or some subtlety.
[12:52:52] <dcaro>	 yep, I'm unaware also of the relations that might exist between other objects that might need updating
[12:53:00] <topranks>	 Is codfw fairly safe to attempt it? We’re ok now so let’s not do anything today
[12:53:32] <topranks>	 yeah exactly.  there is no runbook for setting up from scratch is there?
[12:53:48] <topranks>	 if we seen the order things were added in a fresh setup we could probably work it out
[12:53:48] <dcaro>	 not that I know of, I'll look for it
[12:54:23] <dcaro>	 codfw is for testing, so yes, that's safe, now, others are testing too xd, and now it's working, so I'd wait for monday, spend some time today looking for docs and such maybe
[12:54:46] <topranks>	 yep there is no panic
[12:55:35] <topranks>	 it’s somewhat fragile, if cloudgw keepalived switches, but no reason that will happen
[13:09:07] <dcaro>	 I'm not finding any wiki/task in which the openstack configuration is detailed, or has the steps to do it from scratch :/
[13:27:41] <topranks>	 :(
[13:36:37] * taavi afk for a bit
[14:57:22] <dcaro>	 dhinus: andrewbogott are you playing with cloudlbs?
[14:57:30] <dcaro>	 there's two alerts
[14:57:37] <dcaro>	 summary: WARNING check_alive recent restart 267s
[14:58:15] <taavi>	 I merged a haproxy patch
[15:01:13] <dcaro>	 ack
[15:07:43] <dcaro>	 calling it a week :), cya on monday o/
[15:31:12] <andrewbogott>	 topranks: should we expect codfw1dev networking to mostly work or mostly not work?  We're seeing some bad rabbitmq behavior but not sure if it's a network issue or not
[15:32:21] <topranks>	 I’m away from my desk but it should be ok I think
[15:32:52] <topranks>	 was no issue when I left earlier
[15:36:24] <andrewbogott>	 ok, thanks
[16:20:02] <andrewbogott>	 dhinus: the problem really is some variant of that policy rule because when I make it empty ('allow all') things work. So probably I just need to sort out the syntax or something
[16:22:58] <andrewbogott>	 taavi: Thank you for the doc updates!  Do you want to wait for that last haproxy patch to land before I send the announcement about object storage?
[16:25:12] <dhinus>	 andrewbogott: good, that doesn't seem too bad
[16:25:23] <andrewbogott>	 weird but not bad
[16:25:45] <dhinus>	 yes agreed
[16:26:24] <dhinus>	 andrewbogott: if you want to look at something very different, this is an interesting bug T348668
[16:26:25] <stashbot>	 T348668: Trove instances not being created or restarted with configuration group applied - https://phabricator.wikimedia.org/T348668
[16:27:01] <dhinus>	 I thought I found a fix, but it looks like it's still not ideal because my fix ends up creating a non-utf8 database apparently
[16:27:09] <andrewbogott>	 Oh, that :(  I will try to look later on, will likely require an upstream change
[16:28:02] <andrewbogott>	 welp, I just applied an exact repeat of the change you watched me try and now it works
[16:28:09] <andrewbogott>	 I guess I'll puppetize and watch it break again
[16:30:39] <andrewbogott>	 dhinus: https://gerrit.wikimedia.org/r/c/operations/puppet/+/965779
[16:31:01] <dhinus>	 haha maybe there was a typo we didn't notice
[16:31:21] <andrewbogott>	 maybe!
[16:31:32] <dhinus>	 +1d
[16:36:21] <taavi>	 andrewbogott: I think I landed all of the patches already. I think we're good to go live, although I'd appreciate if I could review the announcement before sending it to cloud-announce
[16:37:27] <andrewbogott>	 taavi: https://etherpad.wikimedia.org/p/Gwods9UzuoEMR79NM8nl
[16:37:30] <andrewbogott>	 feel free to edit!
[16:46:08] <taavi>	 andrewbogott: tweaked a bit, I'm happy with it now
[16:46:20] <andrewbogott>	 great, I will send shortly. Thank you!
[16:48:53] <andrewbogott>	 and of course I sent the email from my personal email rather than work email. I guess we'll see if mailman bounces it
[16:51:48] <andrewbogott>	 I also just now added a brief disclaimer to our docs: https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide#Data_persistence
[16:55:02] <dhinus>	 should we link that page from the sidebar and/or some other Help page?
[16:55:13] <taavi>	 I'll add it to the sidebar
[16:56:36] <dhinus>	 thanks!
[16:58:39] <taavi>	 done, as a new "managed storage" section. (and moved Trove there from the 'instances' section since Trove isn't something you add to an instance, but a separate service)
[16:59:27] <andrewbogott>	 sgtm
[17:38:59] <dhinus>	 lgtm!
[17:44:14] * dhinus off