Repository navigation
Redundant VRouter guest network on wrong interface #3179
Description
Activity
@DennisKonrad Can you post the contents of /etc/cloudstack/ips.json on the VR.
@fmaximus Thank you for helping. This is the content of the ips.json:
{
"eth0": [
{
"add": true,
"broadcast": "169.254.255.255",
"cidr": "169.254.2.133/16",
"device": "eth0",
"gateway": "",
"netmask": "255.255.0.0",
"network": "169.254.0.0/16",
"nic_dev_id": "0",
"nw_type": "control",
"one_to_one_nat": false,
"public_ip": "169.254.2.133",
"size": "16",
"source_nat": false
}
],
"eth2": [
{
"add": true,
"broadcast": "xxx.xxx.xxx.xxx",
"cidr": "xxx.xxx.xxx.xxx/26",
"device": "eth2",
"first_i_p": true,
"gateway": "xxx.xxx.xxx.1",
"netmask": "255.255.255.192",
"network": "xxx.xxx.xxx.0/26",
"new_nic": true,
"nic_dev_id": 2,
"nw_type": "public",
"one_to_one_nat": false,
"public_ip": "xxx.xxx.xxx.56",
"size": "26",
"source_nat": true,
"vif_mac_address": "1e:00:93:00:00:9a"
},
{
"add": true,
"broadcast": "10.32.64.255",
"cidr": "10.32.64.45/24",
"device": "eth2",
"gateway": "10.32.64.1",
"netmask": "255.255.255.0",
"network": "10.32.64.0/24",
"nic_dev_id": "2",
"nw_type": "guest",
"one_to_one_nat": false,
"public_ip": "10.32.64.45",
"size": "24",
"source_nat": false
}
],
"id": "ips"
}@svenvogel Are you using kvm ?
@ustcweizhou Yes we do
@DennisKonrad What I can see in the ips.json, is that mac address is missing.
The lookup uses the mac address to find out the device id.@fmaximus @svenvogel
Hi ,
does this mean when the code that's responsible for updating the "ips.json" would also insert the mac address this could fix the problem?I searched for the ips.json in the code but wasn't able to locate the code responsible for it.
Any hints are much appreciated@DennisKonrad
could you provide more information ?
(1) which cloudstack version ?
(2) Do you use official systemvm template ?some other comments
(1) eth1 should be public ip, and vpc tier ip will be attached to eth2/eth3, etc
(2) ips are saved into ips.json in systemvm/debian/opt/cloud/bin/merge.py
(3) restart VPC with cleanup might fix your problem.@ustcweizhou
info
(1) we are running more or less the master on the agents and mgmts
The systemvm template is built from master too but from december 2018
(2) There are no modifications done to the systemvm templateother comments
(1) yes, eth1->public, eth2++ -> guest networks, VRRP->lowest guest nic
_!! Do you think the problem is, that the public ip is on the wrong interface when using redundant vrouters? (And not the guest ips as I thought...) _(2) I tried to read through this but got lost. It's really hard to see from where some functions are supposed to be called. Is there someone who knows this code?
(3) I will try to do that today@DennisKonrad
when apply a new configuration in VR, it will create a file in VR and execute the update in VR.
see code in ./core/src/com/cloud/agent/resource/virtualnetwork/VirtualRoutingResource.java belowprivate ExecutionResult applyConfigToVR(String routerAccessIp, ConfigItem c, Duration timeout) { if (c instanceof FileConfigItem) { FileConfigItem configItem = (FileConfigItem)c; return _vrDeployer.createFileInVR(routerAccessIp, configItem.getFilePath(), configItem.getFileName(), configItem.getFileContents()); } else if (c instanceof ScriptConfigItem) { ScriptConfigItem configItem = (ScriptConfigItem)c; return _vrDeployer.executeInVR(routerAccessIp, configItem.getScript(), configItem.getArgs(), timeout); } throw new CloudRuntimeException("Unable to apply unknown configitem of type " + c.getClass().getSimpleName()); }json files will be created in server, then scp to VR through 169.254.X.X/port 3922 on VR.
then execute /opt/cloud/bin/update_config.py in VR
json file will be processed in following partdef process_file(): logging.info("Processing JSON file %s" % sys.argv[1]) qf = QueueFile() qf.setFile(sys.argv[1]) qf.load(None)QueueFile is defined in /opt/cloud/bin/merge.py
ips.json will be updated if json file contains new ip.@fmaximus does the info from @ustcweizhou help for a deeper look from you? maybe you have the time.
I found some more time looking into this. @ustcweizhou Do you also think the problem originates in
The root cause seems to be that the interfaces (eth1, eth2, eth3) share the same mac address.
I'm not really sure where this originates. Somehow when the redundant router is created and the nics are added all of them get the same mac.
The "createRedundantAssociateIPCommands" seems to assign the ips to the wrong interface then.@ustcweizhou Can you give me a hint where in the code new vrouters get their nics?
53 remaining items
@weizhouapache I'll build and test it for OvS today. Anything you want me to test?
there is a lot of text here. Do we have plans for 4.13 @weizhouapache @DennisKonrad @svenvogel ?
@DaanHoogland what's the plan for 4.13.1 and 4.14 ?
sorry I came back from holiday today and missed some discussion.@weizhouapache Hi ... back from holiday. i dont know whats tested. there are things from your side they needs to be tested so that we come ready to a PR?
@weizhouapache ping 😄
@svenvogel I will create a PR next week.
I did not fully test it. Hope it will be good :-D@weizhouapache Thanks. Wei ...i think so 😁 we will test it again its maybe easier but what i saw was very good!
@weizhouapache Hi, sorry for the long delay. The last time I tested this I couldn't find any problems. The two setups I tested (KVM with Linux Bridge and KVM with OVS) looked very good
@DennisKonrad i read in your text above we can close this?
@DaanHoogland I would like to wait for the PR of wei to be tested and merged. If the community accepts it then we can close the issue.
This is still an issue for me. Non redundant routers work fine, redundant routers do not.
@DennisKonrad @svenvogel @DaanHoogland
I am testing my changes. If everything goes fine I will create pr tomorrow.Reacted by Rohit Yadav@weizhouapache that sounds great. Big thanks! 👍
solution merged with PR #3847
ISSUE TYPE
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION
Standard cluster deployment regarding this issue. No changes made to sytemvm or the offerings involved.
OS / ENVIRONMENT
Seems to work only with OvS bridges.
KVM with OvS
Projects with VPCs
SUMMARY
Redundant vRouter Offering doesn't work because guest network is on wrong interface in vRouter
STEPS TO REPRODUCE
EXPECTED RESULTS
The guest network should be added on eth1 because this is the interface where both vRouters share the MAC address.
ACTUAL RESULTS
Redundancy state of the redundant routers changes from UNKNOWN to FAILED because conntrackd and keepalived cannot establish the VRRP redundancy between the two vRouters.