OMCH-SDN: An Overlay multi-controller Hypercube-based topology for Software-defined Networks

— Multi-controller architectures have become very necessary for software-defined networks to provide more efficiency, scalability, flexibility and security. However, the controllers must be interconnected in a way that aims to improve the overall performance. In this paper, we propose the OMCH-SDN, an Overlay Multi-Controller Hypercube-based topology to interconnect controllers by using a Hypercube pattern to take advantage of its various mathematical characteristics. The proposal for this research is evaluated through simulations of a real overlay network. The results show that the OMCH-SDN model shows better results concerning the synchronization delay, the throughput as well as the end-to-end latency comparing to a flat-based SDN multi-controller topology.


Fig. 1. The SDN distributed control plane architecture
In an SDN classic flat-based multi-controller topology for example, each controller is connected to two neighbors, except the edge ones, which are linked with just one controller. This topology interconnects the controllers among one level; it is suitable for designs with a few number of nodes. However, when the network scales to a certain number of controllers the time response between far nodes becomes too long.
As we explained this type of topologies present problems that are directly affecting performance and scalability.
For this reason, we propose in this paper, the use of an n-dimensional hypercube because of its advantageous mathematical properties that can be used to enhance the performance and increase the scalability. In fact, it has been deployed in many works, and demonstrates in the past its usefulness and its efficiency. For example, in the networking field, we find HyperCuP [14], which suggests a Hypercube-based peer-to-peer network introducing an efficient method for broadcasting and searching, or BlueCube [15], which offers a Bluetooth network based on a Hypercube, to support parallel computing and ensure fault tolerance. The Hypercube has been proposed in the context of SDN to connect between nodes in the underlying network [16], which is not the case in this paper.
Henceforth, we propose in this paper an Overlay Multi-controller Hypercube-based Software-define Network (OMCH-SDN), which consists of following a Hypercube pattern to connect between the controllers of the control plane using virtual tunnels. We consider this solution as an overlay solution because the controllers can be connected via the OMCH-SDN model independently of the underlying network, which means that controllers can be dispersed around the globe and still able to form the OMCH-SDN.
The remainder of this paper is organized as follows. In section 2, Material and Method, we first recall the definition of a Hypercube, its main properties and their benefits, and then, we describe our OMCH-SDN model. After that, we present our simulation description, by presenting the simulation Testbed, then the performance indices, the synchronization delay, the throughput and the end-to-end latency, which we are going to measure. In section 3, Results and Discussion, we present, discuss and analyze the simulation results. Finally, in Section 4, we give a Conclusion.
II. MATERIAL AND METHOD In this section, we first present the hypercube and its mathematical characteristics, then explain how we will build the OMCH-SDN model.

A. The n-dimensional Hypercube and its proprieties
A hypercube is one of the most efficient ways used to interconnect between objects to improve their interaction performance in many fields because of its different mathematical characteristics. In graph theory, the n-dimensional Hypercube graph ( n H ), with a total number of n 2 nodes, and Hence it is better than a flat topology with n 2 nodes, where the longest path is n 2 -1. represented in an n-bit format. A Gray Code is used to represent each number from 0 to n 2 -1 in a binary form of length n, with the condition that adjacent numbers need to have just one bit that differs in their Gray Code representations. In other words, in a n H , two nodes are directly connected if their binary representations differ with just one bit. This property will help us to assign to each controller node a Gray Code sequence, which will permit us to build our OMCH-SDN topology.

B. The OMCH-SDN model
The OMCH-SDN is an SDN network that has a control plane that contains multiple controllers connected following the hypercube pattern using virtual tunnels. Additionally, the OMCH-SDN is an overlay network, which means that the controllers can be at different locations in the world since the virtual tunnels can be created using routable public IP addresses. In other words, two controllers can be adjacent even if they need to go through multiple routers and switches in the underlying network.
The OMCH-SDN is constructed, using Gray Code sequencing system and basic interaction between the controllers and using the GRE (Generic Routing Encapsulation) [17], which is tunneling protocol that can be used to connect two devices remotely.
The OMCH-SDN constructing process has three steps: Step 1 is the Setting up process where we have c controllers, each one has a routable IP address, and we give to each controller an environment variable with its binary representation (Gray Code sequence), and another environment variable that represents its ID.
Step 2 is The Solicitation process where each controller will send the other controllers a message with its own Gray Code sequence and its ID, looking for those that have their Gray Code string differs with just one bit. Then, create for them a matching number of locally GRE interfaces to attach the GRE tunnels later.
Step 3 is The GRE Tunnels Forming process where each controller will build the GRE tunnels using the GRE interfaces and the routable IP addresses with its neighbors.
After that, the OMCH-SDN constructing process is complete. But before that, even though controllers will have routable IP address, which means that they can reach each other's, when we will create the GRE tunnels, we will assign new private IP addresses to theses GRE interfaces, so we need a mechanism to enable routing. For this reason, we can use for example Quagga [18], a routing framework that can be employed in Linux systems to provide basic and advanced routing between various Linux servers.
To understand how the OMCH-SDN constructing process works, we will show you a use case for a hypercube of dimension k=3.
In a three-dimensional hypercube, we have eight nodes (controllers) and twelve links. Each node can be represented in a Gray Code sequence of three bits. To start the OMCH-SDN constructing process, we should follow the steps already explained.
First, the setting up process where each controller will have a routable IP address, and two environment variables (binary representation + ID). Second, each controller will send its binary presentation to other controllers aiming to find those that differ with just one bit. In this case, each controller will find three adjacents. Therefore, it will create three GRE interfaces that will be used to form the GRE virtual tunnels inter-controllers. Finally, each controller will build three GRE virtual tunnels to establish the OMCH-SDN model.
The following table summarizes the two final steps, while showing the GRE interfaces IP addresses as well as the controllers that will be connected to each controller within this hypercube-based topology: The final illustration of the OMCH-SDN of dimension three will look like the showing in figure 1:

C. Simulation description of OMCH-SDN model
In this subsection, we are going to describe the Testbed, which is built on the top of a real overlay network, present the performances indices that we are going to measure, which are the synchronization delay and the throughput inter the edge controllers.

1) Testbed description:
Our Testbed is using a Hosting Service called ATLENTIC.NET [19] to which we have subscribed to perform this simulation. It provides us with a cloud based environment that is billed by the hour.
To build our real overlay network, we are going to provision Ubuntu Linux servers in the USA-EAST-2 region in New York City. Each server has 2 CPUs and 4G of RAM. Additionally, each server is reachable on layer three connectivity to the other servers via the Internet.
In order to evaluate our proposition, we build our OMCH-SDN for dimension k = 3, which has eight controllers, where each one is up and running on one of our Linux servers that we have provisioned earlier.
We use the open source project, OpenDayLight [12] as an SDN controller, the Beryllium release, because it supports multiple controllers, and has an engaging community that discusses, supports and contributes to this project. OpenDayLight supports an SDN-enabled network with multiple controllers, by building a logically centralized architecture as a cluster which is highly consistent thanks to the continuous synchronization inter-controllers mechanism.
Finally, we attach to each controller to a simulated underlying network generated by Mininet [20] with a particular number of nodes (OVS switches [21] + hosts), we take for example 9 underlying nodes (3 OVS switches, where each switch is connected to 2 hosts) by controller, in total 72 nodes.
The following figure summarizes the testbed for a hypercube-based topology: Fig. 3. The Testbed for a hypercube based topology of dimension 3 with 8 controllers We will also similarly build a flat-based topology with same number of controllers and underlying nodes, to compare with, like you see in the following figure: Next, in the following subsection, we are going to present the performance indices that we will measure.

1) Performance indices:
When evaluating a multi-controller architecture, in particular, the logically centralized which needs a strong synchronization, an important performance index is the synchronization delay, which measure the time consumed to declare that a neighboring controller is aware of an event that was generated on some other controller. For this reason, we are going the measure the synchronization delay between the farthest controllers in the case of 8 controllers' scenario, in a hypercube-based topology and in a flat topology.
Then, we will measure the throughput also between the farthest controllers in our topology. Taking into consideration that the maximum throughput offered by our Testbed is 600 Mbits/s, we would like to know how the OMCH-SDN model will react, and what specific value will reach.
After that we finish by measuring the end-to-end latency. We use Wireshark [22] to compute the synchronization delay, which is a software that permits to see what is happing in the network at a microscopic level.
While, we use Iperf [23] to measure the throughput, which is a tool used for active measurements of the maximum achievable bandwidth in a network and others.
Finally, we measure the end-to-end latency using the Ping utility.

III. RESULTS AND DISCUSSION
In this section, we are going to present and discuss the simulation results, by showing the simulation results of the synchronization delay and the throughput.

A. The synchronization delay
When we lunch the generated Mininet underlying networks inside each OpenDayLight controller, which is already a member of the cluster, we go through two phases, the switching events building, which takes a just few seconds, then the watching phase, which is the lasting phase, where the cluster synchronization happens each time something is changed, added or removed in the network in addition to maintaining a continuous consistency checking to ensure high consistency among all controllers.
In the following figure, we measure the synchronization delay between the edge controllers within the watching phase, while we add, remove or update hosts and flow entries to their underlying networks.  Figure 5 shows the synchronization delay inter edge controllers during a segment of time of 60 seconds, while adding, removing and updating hosts and flow entries to the underlying networks.
We notice globally the synchronization delay for the flat-based topology goes up to 46 ms, while it doesn't go more than 38 ms in the hypercube-based topology. When digging deeper, we see that some values that go beyond 30 ms, however, the synchronization delay for the most of the time is under the 1.5 ms for the hypercube-based topology, and goes between 9 ms and 2.5 ms for a flat-based topology.
When we sum up the synchronization delay for both topologies, we found that the hypercube-based topology is faster 61.88% than the hypercube-based topology.
It is so clear, why we had such an improvement, because in the case of a hypercube based topology, to go from one edge controller to another, we go through 3 controllers, while in the flat based topology we go through 7 controllers. B. The throughput In the following, we measure the throughput between the edge controllers within the watching phase for a duration of 90 seconds. Fig. 6. The throughput inter edge controllers regarding a hypercube-based and a flat-based topology Figure 6 shows that the throughput inter-controllers for a hypercube-based topology is approximately the twice the throughput for a flat based topology.
When measuring the sum of values to found the percentage of improvement, we have found that the hypercube-based topology optimizes the flat-based topology's throughput up to 153%.
Additionally, knowing that that maximum throughput offered by our Testbed from Atlentic.net is 600 Mbits/s, the hypercube-based topology can reach up to 66% of it, which can be considered very efficient. C. The end-to-end latency In the following, we measure the end to end latency using a simple ping, for both hypercube based and flat based topologies. Fig. 7. The end-to-end latency inter edge controllers regarding a hypercube-based and a flat-based topology Figure 7 shows that the end to end latency for a hypercube-based topology is lower in comparison to a flat based topology.
In addition, we have found that the end-to-end latency for the hypercube-based topology is 59.84% faster than the flat-based topology.

IV. CONCLUSION
SDN multiple controllers are necessary to enhance scalability, security and performance. The way we connect between these controllers may affect the overall performance of the SDN based network.
In this research paper, we have implemented and evaluated the OMCH-SDN topology, which is a Hypercubebased way that we have proposed to interconnect between multiple controllers in an SDN environment. The results demonstrated that the OMCH-SDN model has favorably and significantly succeeded to enhance many network performance measurements, like the synchronization delay, which become more 62% faster in comparison to a flat-based topology. In addition, the throughput has been improved and was able to use more of the available throughput.
In general, the OMCH-SDN model has helped to improve the overall performance of a logically centralized multi-controller SDN network.