TSAN : Backbone Network Architecture for Smart Grid of P . R China

Network architecture of any real-time system must be robust enough to absorb several network failures and still work smoothly. Smart Grid Network is one of those big networks that should be considered and designed carefully because of its dependencies. There are several hybrid approaches that have been proposed using wireless and wired technologies by involving SDH/SONNET as a backbone network, but all technologies have their own limitations and can’t be utilized due to various factors. In this paper, we propose a fiber optic based Gigabit Ethernet (1000BASE-ZX) network named as Territory Substation Area Network (T-SAN) for smart grid backbone architecture. It is a scalable architecture, with several desired features, like higher coverage, fault tolerance, robustness, reliability, and maximum availability. The use case of sample mapping the T-SAN on the map of People Republic of China proves its strength to become backhaul network of any territory or country, the results of implemented architecture and its protocol for fault detection and recovery reveals the ability of system survival under several random, multiple and simultaneous faults efficiently. Keywords—Smart Grid; TSAN; 1000BASE-ZX ethernet; backbone architecture


I. INTRODUCTION
The generated data from different sources in a smart grid system is enormous.This data might contain meter readings, real-time price updates, sensor data or other control information.To enable the smart substation system for the exchange of this much huge data is the most critical part of communication infrastructure in smart substations architecture.Though there is no de facto networking standard of smart grid available [1].The current implementation has the hybrid approaches for consideration, in which each technology carries its own weakness and strengths.Thus an entirely new networking system is needed for interconnected substations [1].The communication network of an interconnected substation system demands few essential features, such as Reliability, Acceptable Response delay, Scalability, Fault Tolerance, high availability, Wide coverage, and Security.

A. Reliability
In real-time systems, the communication linkage over a wide area should be based on reliable backbone architecture to enable the timely exchange of messages and commands between the nodes.There could be a number of reasons why a network fails to achieve time critical communication such as; time-out when message delivery delayed because of fault detection and recovery process took longer to resolve, assembling delay, failure of routing protocol and resources failure when any of the responsible hardware resources such as links and other communication devices encounters a physical failures.

B. Acceptable Response Delay
According to IEC 61850 standards, there are three kinds of control messages exchanged in smart grid communication; Generic Object Oriented Substation Event (GOOSE) deals with critical information such as warning and control signals.
Manufacturing Messaging Specification (MMS) used to transfer the substation status information, and Sample Measured Value (SMV) transfer power line current and voltage values measures.GOOSE and SMV are time critical messages need to arrive in < 4ms [2].

C. Scalability
As per statistics of shared public data by World Bank the power consumption in China has grown from the Year 2004 to 2014 from 1585.83 kWh to 3927.04 kWh, Fig. 1 shows the increasing trend of electric power consumption in last 10 years, this growth would be increasing with increase of population and growth of electronic/electrical devices, private power generation plants and other consumption and generation resources in coming future.This huge growth in the field of smart grids needs to have a scalable architecture to adjust and merge new changes flawlessly without disturbing the existing network.www.ijacsa.thesai.org

D. Fault tolerance
The backbone shall have the ability to tolerate each kind of rising faults in the network, and these faults can be broadly categorized into two: Hardware faults and software faults; Hardware faults are those in which hardware equipment are involved like Network Interface Cards, Routers, Switches, network cables, loose connections, and broken/dead systems, Software faults are; congestion on one line, drop of packets due to interference and suspended systems [3], [4].A system should be able to cope up both kinds of hardware and software faults.

E. High Availability
It is essential for smart grid implementation to ensure the availability of power and communication to the consumers, especially when dealing with issues like latency and security [5].And the availability is dependent on the robust connectivity of both power and communication networks and timely data exchange between consumer and service provider entity.

F. Wide Coverage
The power system doesn't require serving a particular place, an entire country or territory should be covered by a Wide Area Network to serve the scattered locations of the power grid systems.Wide Area Network should be able to bring the real-time measurement data from substations/customers to the Control Centers situated at distances.

G. Security
Wide area of the electrical system can be scattered over 1000 of miles, thus the communication system should be able to protect itself from potential attacks in the cyber-physical world.Communication mediums like Wireless technology are the most vulnerable candidates to consider for wide area network because of security.
Thus a backhaul network with large data rate is required which satisfies all the stated requirements of Interconnected Substation System and it should also be a cost-effective solution in implementation and maintenance.Currently the implemented system is using a hybrid approach of wireless and wired technologies for providing this interconnection on the backbone of SDH/SONET, these are not available as a standard technology neither they are satisfying all the requirements of the smart grid system, Section 2 further covers the details of all the available technologies and their up and downs.In this paper there are two major contributions to consider: The first one is proposed novel network backbone architecture named as TSAN, it is an inspiration of our already proposed Recursive Scalable Autonomous Fault-tolerant Ethernet (RSAFE) architecture for mission-critical systems [6].The difference in RSAFE and TSAN is that it involves fiber optic communication, implemented on wide range, and the fault-tolerant protocol is enhanced in TSAN than RSAFE.which is based on Gigabit Ethernet standard 1000BASE-ZX (also referred as GigE) transmission using 1550 nm wavelength ranges the distance of 70 km (at least, while some vendors claim the maximum range of 120 km) over single mode fiber optic cable.The proposed backbone architecture is a hybrid approach of star, mesh and ring topologies.It divides the overall land into manageable Regions and Zones despite considering the boundaries of geographical division (town, city, district, province, etc.) assigned by a territory or country.A zone connects all the small and large substations and control centers in the same network using dual star topology with two Ethernet layer 3 switches/routers.Then two or more than two zones combined in a dual ring topology to form a region; similarly, all the parts of a country are divided into zones and regions and then those regional networks are merged with each other using neighboring zones of respective regions.All the nodes in the proposed network architecture are dually connected, where one link serves as the primary path and other as a standby path.This well-connected hybrid topology aims to provide robustness, reliability, scalability, and wide coverage and high availability to the system.
The second major contribution is Fault Tolerant Ethernet Protocol (FTEP), implementation of FTEP is based on two modules fault detection and fault recovery.In first module, all the substations within a zone send an Aliveness Beat Message (ABM) to their respective control centers for indicating their active path.Each ABM is dispatched at the fixed interval of 2ms from all the substations to the control center.The control centers besides communicating the operational commands also update the routing table according to received network status and forward the changes to the neighboring zones.The second module is executed if any fault has been detected by first module (missing of 2 consecutive ABMs), as discussed in proposed architecture above, there are two links to reach a node in a zone, one is primary (operational) and the other one is secondary (standby), fault recovery module enables the standby link and update the link information by making standby path as default communication path in routing table.The results of proposed backbone architecture and protocol demonstrate the ability of the TSAN to fulfill all the needs and provide a basic and complete suite of solution for Smart Grid backbone network.
Rest of the paper is organized as follows.Section II presents the available technologies for WAN in Smart Grids, Section III describes the proposed TSAN for Smart Grids, Section IV is about experimental setup, Section V presents the results and Section VI concludes.www.ijacsa.thesai.org

II. CANDIDATE TECHNOLOGIES FOR WAN IN SMARTGRIDS
Currently, the smart grid Wide Area Network implementation has eight main technologies for consideration: Power Line Communication (PLC), Digital Subscriber Line, Wireless Mesh, WiMAX, Cellular, Space Communication (Satellite Communication), SDH/SONET and Gigabit Ethernet.These all technologies have their strengths and weaknesses as shown in Table I.

H. Digital Subscriber Line
The Digital Subscriber Line or DSL have three systems to offer ADSL, HDSL, and VDSL depending on usage it may offer different range and data rates.But due to its reliability and downtime issue, it couldn't be considered for the backbone of huge and real-time communication architecture.

I. Power Line Communication
This technology could be a good candidate as its network already exist, but its limitation to transfer of signal across the transformer raises the concern also it experiences heavy noise, interference and involve insecurity issue [7]- [13].

J. Wireless Mesh
It's a good candidate; provide a mesh of a multi-hop wireless network.Has a good data rate, can be implemented using 802.11,802.15 and 802.16.But like almost all other wireless technologies this has also a problem with interference and suffer from noise [7].

K. WiMAX
Worldwide interoperability for microwave access is a 4G wireless technology dedicated to the advancement of IEEE 802.16 and series of standards for Metropolitan Area network such as IEEE 802.802.16e.It operates on both licensed and unlicensed frequency bands.However, it's an expensive implementation on a wide range and also requires high power consumption.Furthermore, unfavorable weather conditions can also affect this technology [14], [15].

L. Cellular
It's a mobile communication technology; works on radio signals, the network is made up of several radio cells.One of the main advantages of this technology is that its infrastructure is already available [14].The downside is that the consumer equipment up gradation cost is high and the network is shared with mobile customers, it's not only a security concern but congestion is also expected [16].

M. Space Communication or Satellite Communication
The satellite communication is able to provide global coverage even in the rural areas of the country with a data rate of 1 Mbps.It can provide GPS based satellite monitoring and synchronization of any site.It's a cost-effective solution but the limitation of this implementation is that it suffers severely from the weather conditions, which may lead to long round trips [17].

N. SDH/SONET
Its large data rate and range convinced fiber optic to be a backbone cable.The up gradation and installing new network could be expensive but the quality of service delivered by fiber optic is better than all other existing technologies, moreover, it is immune to noises and used in long run [7].The good side is SDH/SONET provides large and managed network, simple topology, and the resilient ring, which is the most liked and its widely used option.On the downside, the dynamic IP traffic is not optimized here, configured with fixed P2P bandwidth, so the allocated bandwidth is not efficiently utilized and always the unused bandwidth is wasted.Limited topology options like P2P, Linear, and Ring.Inefficient in transferring the multicast traffic, the implementation of the Ring is up to maximum 16 Nodes with coverage of 1200km.SDH/SONET doesn't cover all over the transmission and distribution line communication because of its limited coverage.Thus it may not work as an individual network to cover every corner of a country, the smart grid network would have to depend on any of abovementioned technologies to gather the data from all areas and send it on to the SDH/SONET backbone network [18].

O. Gigabit Ethernet
The Ethernet 1000BASE-ZX Standard is Gigabit over single-mode fiber cable with a wavelength of up to 1550nm, which is able to cover 70 Km distance [19].The limitation of this technology is that it requires entirely new-dedicated cable installation.In comparison with SONET/SDH 1000BASE-ZX could be a good option in several ways: Efficient utilization of bandwidth in P2P and mesh [20].Furthermore, it is highly scalable, and no need to provide conversion for synchronization from different wires, cost-effective, mesh topology offers exceptional utilization of bandwidth.I and discussing pros and cons of PLC, DSL, Wireless Mesh, WiMAX, Cellular, Satellite and SDH/SONET the Gigabit Ethernet could be the potential technology, to provide vast and efficient coverage with good data rate, less interference, less CapEx and OpEX, and moreover an entirely dedicated network.

III. PROPOSED TSAN ARCHITECTURE
In a country, there could be thousands of substations that send and receive information updates every time to each other using communication network.Thus the backbone communication network of a system is an important part to pay attention and develop the strategic network that should be able to survive longer, stronger, secure and scalable.This paper aims to propose network design architecture and an implementation of a fault tolerant protocol that results to achieve a network according to the needs of smart grids.

A. Network Architecture
The proposed design is a Gigabit Ethernet network with a combination of mainly three topologies (star, mesh, and ring) this network is providing reliable communication architecture, which can work well with multiple failures, the TSAN divides the overall territory into manageable zones and regions despite considering the boundaries of geographical division such as; town, city, district or a province assigned by a country government.The geographical divisions are named as Zonal Substation Area Network (ZSAN), Regional Substation Area Network (RSAN) and the combination of these two; Territory Substation Area Network (TSAN).www.ijacsa.thesai.org

B. Zonal Substation Area Network (ZSAN)
One Zone is considered as a locality with multiple substations connected together and reporting the locally available control center(s), their communication network is a combination topology of star and mesh, where each substation serves a particular area feeders and report to the dedicated control center.As shown in Fig. 2, a ZSAN provides a primary path the connection in blue and a standby path doted red connection.Both are used to ensure reliable communication within the network.The standby path here is used only to rescue the condition of failures in a communication network.One ZSAN can connect maximum 64 nodes including large and small substations and the control centers.

C. Regional and Territory Substation Area Network (RSAN and TSAN)
As shown in Fig. 3, a region is a huge area connecting different zones in a dual ring network like SDH/SONET; it is named as Regional Substation Area Network (RSAN).Fig. 3 shows three regions, namely, Region A, B, and C; all of these regions combining to form a Territory Substation Area Network (TSAN).The connection in blue is primary communication path and red is standby path.The redundant rings ensure the intra-region connectivity and redundant linkage with other regions are for reliable inter-region connectivity, where, the size and the number of regions depending on country's geographical features, size, and location.
The proposed network design architecture has several advantages over SDH/SONET: a) The architecture is entirely based on Ethernet, so it's not needed to convert to another communication protocol b) It provides a combo topology (Mesh, Star, and Ring), which makes it more reliable than SONET/SDH, and less expensive (in terms of wastage of bandwidth) which is a common issue of SDH/SONET.
c) It uses COTS products, thus no proprietary hardware needed.
The Fault detection and recovery protocol suit provide the desired results according to IEC 61850 standards.

D. Fault Tolerant Ethernet (FTE) Protocol
The FTE protocol is implemented using Aliveness Beat Messages (ABM), where an ABM is a lightweight Ethernet frame periodically sent by all substations to the dedicated Control Centers in their respective zones for indicating their active connections; on the other hand all the control centers receive ABM and maintain their own copy of updated list of routing table.In a zone there could be more than one control centers, thus one of them is selected as a primary control center (PCC), which perform some additional responsibility of communicating their routing table with the PCC of the zones (left and right zones) in the region as shown in Fig. 2..The FTE protocol working depends on the recipient of ABMs from the all substations in a network to their control centers.We can say that FTE Protocol is divided into 2 different working modules.One is Fault Detection another is Fault Recovery.As shown in Fig. 4(a) where one substation monitors its connections and send ABM to its control center, if any change has taken place in connection information, it will be updated there.The second module of fault recovery is as depicted in Fig. 4(b), which shows a control center is receiving ABMs.In a zone each substation is responsible to send an ABM over a fixed interval of 20ms to their dedicated control center, and if two consecutive ABMs are missed the link would be considered as down and it will change the primary link information with standby link and convey the updated information with all the substations in its own zone and other zones of the same region.www.ijacsa.thesai.orgThe suggested TSAN and the FTE routing protocol satisfies the requirements of the WAN networks in Smart Grids: 1) Reliability: Dually Connected, multipath architecture, so even after random multiple failures a substation can still communicate with rest of the world using alternative standby path(s) 2) Scalability: At any instance of time the new substation or an entire new zone can be added without interrupting the existing network.
3) Fault Tolerance: The proposed protocol provides efficient recovery from faults within specified time limits.
4) High availability: This fault detection and recovery and dually connected network provides high availability of the communication network, and have the ability to adjust with a large number of random faults occurrence, which can be a helpful strategy in disaster situations.
5) Wide coverage: The implementation of proposed scheme is exemplified by mapping the TSAN on the map of the Republic of China in next section.
6) Security: The dedicated network and firewall protection provide maximum security from unidentified intrusions, which is not possible with other shared networks or any of Wireless technologies.

E. Use Case of TSAN -People's Republic of China
The sample mapping is shown in Fig. 5. Depicting the connectivity example of smart grid substation area network, there are 39 zones where each zone may have maximum connectivity option of 64 substations.
In current scenario, if every subnet consists of 64 substations then this overall network is comprised of 2496 large and small substations.For security each zone have their own firewalls to provide intra and inter-network protection.Fig. 6 shows the clean backbone connectivity rings throughout the country for providing communication network in every corner of the country.

IV. EXPERIMENTAL SETUP
The experiment is done by creating a flatbed setup in a lab, where 12 computers forming 3 regional subnets and connected using Fast Ethernet.The regional subnet connects the entire substation dually in a network, and periodically an ABM is sent via each substation to their dedicated control center for showing aliveness of a link.
As shown in Fig. 7, A Region consists of 4 nodes (3 substations and 1 control center) in this scenario, and 2 switches.Where every node is having 2 NICs named as Eth-A and Eth-B and connects to Switch-A and Switch-B respectively in their zone.The control center besides performing its controlling task is also additionally made responsible to collect Aliveness Beat Messages from the all substations and communicate the status updates with other regions by broadcasting a periodic status updates.Table I gives the details of hardware and software components used in experiment.www.ijacsa.thesai.org

V. RESULTS
Several experiments were conducted to validate the functionality of the suggested scheme TSAN. Figure 8, shows our theoretical assumptions of aliveness message and fault detection which is unrealistic when it comes into action as shown in Fig. 9 shows the spikes as variations in number of aliveness messages that reflects the actual time of fault occurrence.As shown in Scenario 1, Fig. 10 the ideal situation is that every substation sends an aliveness message exactly at the interval of every 2 seconds to the control center and if there is a link or device or any kind of failure occurred somewhere it should take 4.4 seconds to detect and recover that fault.
But practically it varies as Scenario 2 and 3 of Fig. 10, the Substation A sends aliveness message to Control Center and got a failure, but the detection took longer than expected time but it cannot exceed the time of three heartbeat messages.Equation 1 defines overall latency in the network.
In ( 1), Time TL is total latency, F Gap is the time interval between frames (ideal time is 2ms), F size , is size of the frame, N www.ijacsa.thesai.orgnumber of nodes in a zone, SW Latency is the operational latency of switch, MD Latency latency involved by used cable medias.
For checking the sustainability of our proposed system we have created 8 different link failures on different time intervals as shown in Fig. 11, these multiple random failures are recovered in a timely manner and the network is made again stable before introducing new fault.As mentioned above, this experiment involves 9 substation PCs and 3 Control Centers PCs, not only substations but also control centers may encounter physical faults in a network, so every substation sends aliveness message to their control center and control center exchanges the status updates with other networks, that exchange is considered as the aliveness signal of a regional area network.
Thus the result in Fig. 11 contains 0 to 8 substations and 9 to 11 control centers.From these 8 failures, 3 failures were control center failures and 5 are the substation failures.And after each fault, the linkage information is successfully updated within a limited time.From (1), three main parameters N, SW Latency and MD Latency are responsible for causing a delay in communication; the resulting spikes can be seen in Fig. 11.And the results shown in Fig. 12 are presenting the simultaneous multiple failures; these are detected and recovered exactly in the same manner as random faults.Every node is still made approachable in the network in the very short duration of time.This proves that our system can tolerate limited multiple and simultaneous link failures, and still, the network is completely connected and approachable from everywhere.

VI. CONCLUSION AND FUTURE WORK
We implemented Ethernet-based network architecture for Territory Area Network of Smart Grid Equipment and a protocol for detection and recovery of physical failures in the network system.The beauty of using Ethernet is it doesn't require extra effort of conversion of the protocol stack during transmission because the Territory Area Network is entirely implemented using Ethernet.We modeled the approach on the map of China considering 2496 Substations and control centers throughout the country to show the applicability of the scheme in big countries.The validity of proposed scheme and protocol is supported by a flatbed experiment, which has 3 zones, each zone with 3 substations and 1 control center in it.The experimental result shows that the system is able to detect and recover faults within acceptable time delay.The system is also able to handle multiple random and simultaneous faults within a specified limit, which is quite sufficient for supporting during rescuing condition of Smart Grid system.We also plan to work on our proposed protocol for reducing the fault detection and fault recovery time in future.

Fig. 1 .
Fig. 1.Growth of electric power consumption of China.

Fig. 11 .
Fig. 11.Multiple random link failures at different time intervals.

TABLE I .
SPECIFICATIONS OF EXPERIMENTAL SETUP