User:Shenh10

When choosing cloud platform such as OpenStack to deploy a computing framework, the main challenge exists in efficiency. To store processing data inside Virtual Machines is neither reliable nor collaborative.

However, deploying a shared filesystem on physical machine leads to bottleneck of network bandwidth at the Neutron node. We promoted a solution achieving better networking performance in such a deploy case. By hacking into OVS and OVS Agent, I have achieved Hadoop's layer2 access to physical HDFS which improved the locality and avoided layer3 router congestion. Furthermore, Docker can also utilize this solution to get switching performance comparing to Linux Iptables forwarding. Based on such optimized platforms of 100+ nodes cluster, I have done a benchmark on how Hadoop behaves when deployed on virtualization platforms( Baremental, OpenStack, Docker)

A set of bash scripts goes here https://github.com/iiisthu/HadoopBenchmark

And a template for deploy Hadoop in Openstack Virtual Machines goes here https://github.com/iiisthu/HadoopTemplate