A Monitoring Model for Hierarchical Architecture of Distributed Systems

—Distributed systems are complex systems and there are a lot of the potential risks in the system, so system administrators need to have some effective support tools for network management. The architecture information of distributed systems is an essential part of distributed system monitoring solutions, because it provides general information about monitored objects in the system for administrators, as well as supports administrator in quickly detecting change of topology, error status or potential risks that arise during operation of distributed systems. The modeling approaches have an important role in developing monitoring solutions, in which they are background to develop algorithms for monitoring problems in distributed systems. This paper proposes an approach in order to model for hierarchical architecture of objects in distributed systems, in which consists of architecture of monitored objects, networks, domains and global distributed systems. Based on this model, a basic monitoring solution for hierarchical architecture of distributed systems is developed and this solution is able to provide administrators more important architecture information such as the topology of hardware components, processes, status of monitored objects, etc.


I. INTRODUCTION
Distributed systems (DS) are complex systems, which have always challenged for system administrator a lot [5,9].A hardware malfunction, a faulty process or an abnormal event occurs on the system may affect other events taking place at different locations in the running environment of system.These symptoms can cause a bad effect on performance and stability of the system, they can also cause of errors of related processes and incorrect results of distributed applications.In order to ensure the effective operation of DS, global system information in general and information of each object in particular is critical issues.Many technical solutions have been researched and developed to support administrators in monitoring the system.Through the survey and review some typical monitoring works such as [10,11,13,14,15,16,17] in paper [4,8], in which we presented in detail the technical details, some advantages as well as disadvantages of these solutions.The survey result on implementation solutions and function of monitoring systems is presented in Table I and II.can be divided into two groups: specific monitoring (SM) and general operations (GM) for monitored object in DS.
 SM consists of monitoring systems that monitor specific issues of monitored objects in DS such as MonALISA, MOTEL, SNMP, etc. SM can be seen as a special monitoring layer to monitor details such as traffic, performance, computing, etc.Most of these solutions in SM are only focused on solving the requirements for specific monitoring issues between www.ijacsa.thesai.orgobjects and have not yet been really interested in the global architecture of monitored objects in DS.For example, ZM4/SIMPLE is deployed to do performance evaluation for and parallel and distributed programs; MonALISA is deployed to monitor and help manage and optimize the operational performance of Grids; etc.
 GM consists of monitoring systems that monitor general operations of the monitored objects in DS such as built-in tools of devices or utilities in OS (Operating System).GM can be seen as a common monitoring layer in which provide abilities to monitor architectures and operations of monitored objects (MO) such as configuration, status, communication, connections, etc.For example, taskmgr and netstat commands are in Windows OS; prstat command is Solaris OS, etc.
Therefore, we can divide monitoring for DS into two basic stages:  The first stage is general monitoring with monitoring solutions in GM, the global architecture information of monitored DS in general and the information about general operations of monitored objects in particular are essential in this stage, because they can support administrator for quickly detecting common errors or error domains that arise during operation of the system [4].
 The second stage is extended survey with monitoring solutions in SM in order to go into more detail in special monitoring information.
Thus, the monitoring solutions in GM are considered as a high level monitoring facilities to monitored DS before using other monitoring solutions in SM to deeper analysis.However, GM are now mainly based on tools (OS, utilities) that developed by device vendors side or operating systems side.These built-in tools have some disadvantages such as discrete monitoring information, independent of each device, etc [4,8], hence the global of DS cannot be solved with these built-in tools.The global architecture should be continued to research and develop more effective, the goal of the paper focus on solving this problem base on modeling for architecture of MO and building hierarchical monitoring entities respectively.
When monitored systems have basic changes about architectures, behaviors and operation environments, the technical solutions must be modified and updated appropriately for new changes and management requirements.As system specification methodology is generally and flexibly, the modeling approach is considered more appropriate for systems that have a lot of changes and the approach is widely used in discrete event systems, computer protocols [1,3,7].In the DS, the modeling approach also achieved some certain results [2,6].The modeling approaches play an important role, in which it is used as a basis layer for algorithm and solution development in monitoring, diagnosing and controlling issues independently.Therefore, the modeling for MO in DS is really necessary, the objective of the paper is based on the research results on DS and set theory [1,4,6], we focus on building a formal model for the hierarchical architecture of MO in DS, in which consists of architecture of monitored objects, networks, domains and global distributed systems.We also present a basic monitoring model for the hierarchical architecture of DS, in which can show DS topology visually as well as the local operations and the communication operations of MOs in the DS.
The paper is organized as follows: In section II, we present architectural model for a MO in DS and the composition operation that allows us to combine many MOs into a composition model, we describe hierarchical architecture of monitored objects in DS.Section III focuses on the modeling solution that is able to monitor the architecture of DS.Finally, section IV concludes with the current work and future perspectives.

II. THE ARCHITECTURE MODEL FOR DISTRIBUTED SYSTEMS
DS consists of many heterogeneous devices such as stations, servers, routers, etc.These devices are considered physical objects in DS and communicate to each other in the system; each device consists of many hardware components such as CPU, HDD, etc. and software components such as processes.These components are associated with information about the corresponding states and behaviors, general operations of MO is described by Fig. 1, they can be divided into two basic parts such as internal partlocal operations and external partcommunication operations [4]. The communication operations: these operations are functions that interact with other objects in the system such as inter-process communication, controlling to interact with management system, etc.These operations are used to communicate with other objects on the system.
All of local and communication operations are based on system resource of MO such as CPU, RAM, I/O, etc. and information of these operations is dynamic in their running process, while system resource of MO is static information.Therefore, architecture of MO will consist of static information of MO and dynamic information of local and communication operations.www.ijacsa.thesai.orgMOs are considered as nodes that are connected according to specific architecture and can perform interactive communication to each other.Hence, Architecture model describes the structure of nodes along with the related information of each node, the link between nodes, message propagation via its port, etc.Based on this information, we can determine the physical structure and the state of the nodes in the system.
From result of research on DS and monitoring systems, we can see that DS consists of many heterogeneous objects and topologies that communicate to each other.With point of view the domain-based management for large scale systems, the multi-level domain has been used to manage for DS [18], in which consists of local object level, network and domain level.The hierarchical architecture of monitored objects in DS can be presented as Fig. 2: status is a function that identify node states in which consist of normal or abnormal status, status(NODES) {S_NOR} or {S_ABNOR}, where: S_NOR is set of normal status such as up, communicating,...; S_ABNOR is set of abnormal status such as down, overload,... event is a function that identify node events such as request, messages,...These events consist of internal (internal_events) and external events (external_events) In order to visually present architecture model, we denote AM for architecture model, nNODES, dDOMAIN, netNETS, LLINKS, {p 1 , p 2 }PORTS.So architecture model AM can be visually described as Fig.  event={local operation events} Therefore, architecture model of monitored object will give us more important information about that object such as local operations (internal operations) as well as communication operations (external operations).Based on this architecture information, we can determine operations, errors or abnormal states that occur in running time of the node.

B. Compositon Model
DS is complex system in which consists of many heterogeneous devices (nodes) and is organized according to hierarchical architecture as Fig. 2. So architecture model of DS will be set of architecture model AM of nodes in system.In order to ensure more efficient to build architecture model of DS, we use composition operation as described here.www.ijacsa.thesai.orgLet AM 1 , AM 2 be architecture model of node 1 and node 2 in system, let || be composition operator (concurrent) for AM 1 and AM 2 .Composition operation is shown in Fig. 4. The architectural model after composition AM 1 and AM 2 is AM_C, and AM_C is also a 9-tuple as expression (2). Where: with internal_events(n 1 ): local events in node 1; internal_events(n 2 ): local events in node 2;  12 : communication events between node 1 and 2 Therefore, composition model AM_C describes operation information of two nodes in which consist operations of each node and communication between node 1 and node 2.
Similar to architecture information of MO, we can easily determine operations, errors or abnormal states of node 1 and node 2 that occur in running time based on elements in the model AM_C.

C. Modelling for Architecture of DS
As we presented in section II, topology of DS can be seen as hierarchical structure consists of many levels such as local object, network and domain level, in which global DS consists of n (n0) domains and can communicate with each other via telecommunication networks, each domain consists of m (m0) heterogeneous networks interconnect to each other, and each the network consists of k (k0) physical devices.All off them can collaborate, exchange and share information to each other.Therefore, the modeling for architecture of DS will be done with four levels: MO  From expression (3)( 9), we see that AM_MO, AM_MS, AM_MD and AM_DS are built from composing architecture model of basic objects.Thus, information of model AM_MO, AM_MS, AM_MD and AM_DS will describe all of system information, operations, links and state information (normal, abnormal, error) of elements in them.For example, related information of any network will describe in expression (5), so NODES MS will describe information of all MO in a network because NODES MS = NODES 1  NODES 2 … in which consists of system information, operations and error or abnormal information of all MO.Communication ports PORTS MS will display all of ports of objects in the network, because PORTS MS =PORTS 1  PORTS 2 … .Therefore, in order to determine error or abnormal states of network according to AM_MS, we only NODES_A MS , because

A. The Teachnical Base and Basic Monitoring Solution
The objective of the monitoring system is observation, collection and inspection information about the operations of the hardware and software components, communication events of MO.This information supports actively in system management.
The general monitoring architecture can be divided into 3 parts as Fig. 5. Monitored Object (MO) consists of independent objects such as switches, routers, workstations, servers, ... these objects have their own hardware and software resource.In order to describe architecture information of MO at time t, we use function monitoring_info(MO,t).
Monitoring Application (MA) is designed to support for the management Objects (administrators or other management agents).MA entity interacts with monitoring entity to support the generation of monitoring requirements and present the results of monitoring are measured from monitoring entity.

ME (Monitoring Entity
) is designed to instrument the monitored objects, the instrumentation information of the system will be processed to generate the corresponding monitoring reports and send to MA.In order to describe result of monitoring entity ME at time t, we use function result_ME(ME,t).Thus, monitoring result of ME at time t for MO can be expressed as follows:

 result_ME(ME, t) = monitoring_info(MO, t) (10)
The monitoring system for DS consists of more MEs and MAs, they are not fixed and independently operate on each domain of DS, and monitoring information is exchanged between the MEs and MAs by message passing.
With the hierarchical architecture model of DS is presented as the previous session, hierarchical architecture of DS consists of four levels such as MO, network, domain and global DS.In order to collect the architecture information of DS, monitoring entities are designed in accordance with the hierarchical architecture of DS and we use four monitoring entities to monitor hierarchical architecture of DS:  The monitoring entity ME_MO for object: ME_MO observes and collects the architecture information of MO.Because architecture model of MO is expressed as AM_MO in (3), the monitoring result of ME_MO at time t can be expressed as follows:

 result_ME(ME_MO, t) = monitoring_info(AM_MO, t) (11)
 The monitoring entity ME_MS for network: ME_MS observes and collects the architecture information of a network.Because architecture model of a network is expressed as AM_MS in (4), the monitoring result of ME_MS at time t can be expressed as follows:

 result_ME(ME_MS, t) = monitoring_info(AM_MS, t) (12)
 The monitoring entity ME_MD for domain: ME_MD observes and collects the architecture information of a domain.Because architecture model of a domain is expressed as AM_MD in ( 6), the monitoring result of ME_MD at time t can be expressed as follows:  result_ME(ME_MD, t) = monitoring_info(AM_MD, t) (13)  The monitoring entity ME_DS for distributed systems: ME_DS observes and collects the architecture information of DS.Because architecture model of DS is expressed as AM_DS in ( 8), the monitoring result of ME_DS at time t can be expressed as follows:

 result_ME(ME_DS, t) = monitoring_info(AM_DS, t) (14)
From expression (11÷14), the monitoring system for hierarchical architecture of DS will be set of monitoring entities {ME_MO, ME_MS, ME_MD, ME_DS} that are designed as Fig. 6. www.ijacsa.thesai.orgThe monitoring entities ME_MO will be installed on all of MO in DS, they observe and collect the architecture of MOs, and supply monitoring reports to network monitoring entity ME_MS.ME_MS runs composition operation in order to synthesize monitored information from ME_MOs in the same network and supply network monitoring reports to domain monitoring entity ME_MD.The operation of ME_MD and ME_MS has also run into similar processes as above.The monitoring implementation of ME_MO is designed as Fig. 7. Therefore, the collection and composition process for building the architecture of DS is implemented as following sequence: The collection and composition process of hierarchical monitoring architecture are described detail in Fig. 8.At each level of monitoring entities (MO, network, domain, global DS) will collect full monitoring information of their monitored objects.First level, ME_MO collects and processes all of monitoring information of components such as Processes, CPUs, etc. Second level, ME_MS composes all of monitoring information of MOs in the same network and creates the monitoring report for architecture of this network.Third level, ME_MD composes all of monitoring information of networks in the same domain and creates the monitoring report for architecture of this domain.Fourth level, ME_DS composes all of monitoring information of domains in DS and builds the monitoring report for architecture of DS.
In order to analyze the architecture information of DS, the sequence of steps is implemented as follows: For example, suppose that distributed system CDS consists of two domains {d 1 , d 2 }, each of domains contains one network: net 1 in domain d 1 , net 2 in domain d 2 , network net 1 consists two nodes {n 1 , n 2 }NODES, and network net 2 consists three nodes {n 3 , n 4 , n 5 }NODES.www.ijacsa.thesai.orgAfter the step ME_DS composes all of monitoring information for architecture of DS, we have all of architecture information of CDS that is expressed by the architectural model AM_DS in (9).Therefore, the architecture of CDS is analyzed as follows: From above architecture information, the hierarchical architecture of CDS is presented as Fig. 9.In normal case, all of monitored objects {n 1 , n 2 , n 3 , n 4 , n 5 } are running smoothly, set of information of objects in CDS contains in NODES CDS which consits of system resource information NODES_S CDS , information about operations NODES_O CDS and error information NODES_A CDS .Because the CDS has not any error, NODES_A CDS has not any description.Suppose that objects n 5 is down or overload, then NODES_A CDS contains down state or overload state of n 5 .Base on NODES_A CDS , we will monitor all of errors or abnormal of CDS.
The connection and communication information of objects in CDS such as LINKS CDS , PORTS CDS , port, communication, etc will support us in building algorithms to display network visualization which consists of communication operations and link diagrams of nodes, networks and domains.

B. The Initial Experimental Results
Based on the model is presented in the previous sections, we designed a MCDS (Monitoring for Complex DS) system that consists of a set of monitoring entities (ME_MO, ME_MS, ME_MD, ME_DS as Fig. 6) for monitored objects, group of monitored networks, monitored domains and global system.The goal of MCDS is that monitor the architecture and operations information of devices on the VMSC3 system (a network system of VMS company at Vietnam), in which the architecture of monitored system can be displayed in hierarchical architecture as Fig. 2; operations information consists of local and communication operations (as Fig. 1) of monitored objects in VMSC3 system such as process, communication ports, etc.
The initial experimental results are shown in Fig. 10, in which presents some monitoring forms of MCDS such as group of forms about basic architecture of objects and group of objects in VMSC3, as well as general description information about objects in system such as devices name, IP,...; group of forms about the communication and local operations information of monitored objects such as system information (descriptions, locations, OS…), hardware information (Cpu, Ram, I/O, …) and information on the operations of the processes, status, communication, etc.This information is collected by ME_MO and will be used to send to other monitoring entities (ME_MS, ME_MD and ME_DS) by the message passing mechanism.In order to evaluate this monitoring model for hierarchical architecture of distributed systems, we use some notation such as M our for our model; M GM for monitoring models is mainly based on tools (OS, utilities).Some evaluations as follows:  Monitoring presentation: Because built-in tools only run object itselft or by remote, so discrete monitoring information, independent of each device.Therefore, M GM focuses on presenting monitoring information directly of objects MO in DS, it is only local part of DS.M our presents monitoring results as the hierarchical architecture such as objects, networks, domains and global DS, so the presentation results consist of local part and global system, it provides an overview on monitored DS for administrators and is more appropriate for architecture of complex DS in the practical environment.
 Monitoring function: Solutions in SM group are only pay attention to solve special monitoring issues between MOs (computing, traffic, etc).M GM focuses on general operations of MOs (devices, components) in DS, however this solution provides discrete monitoring information and have some disadvantages that we presented in [4,8]

IV. CONCLUSION
The modeling has an important role in the development of efficient algorithms for the monitoring problems in DS.This paper proposes a modeling method for the basic architecture of objects in DS, the monitoring solution for hierarchical architecture of DS.With the proposed models, we develop the MCDS solution that supports administrators for monitoring information visually such as the DS topology, the operations and status information of objects in the system, etc.Based on the monitoring entities, we easily develop extensions for these monitoring entities to provide complete online architecture information that effectively support for administrators, as well as allow storing monitoring data into database for the synthesis, evaluation and analysis of historical monitoring data later.This information is actively useful for the appropriate management decisions and controlling actions the monitored system.
In order to effectively deploy the monitoring solution for the distributed systems, we continue investments to complete the solution and optimize for monitoring algorithms, the dynamic management model and effective communication model for monitoring entities, as well as the analyzing techniques that optimize the computations for the large number of monitoring information in the large-scale systems.

Fig. 1 .
Fig. 1.General operations of the monitored object  The local operations: these operations include processing, computing, resource requirements for process computations.The operations are locally performed within that object and use system resource such as CPU, RAM, etc. in running time.

Fig. 2 .
Fig. 2. The hierarchical architecture of objects in DS Therefore, the architecture model of DS and monitoring model are presented in hierarchical architecture as Fig. 2 in order to deploy a suitable monitoring solution.A. Architecture Model of MO Let AM be an architecture model of monitored node, the AM is a 9-tuple and expressed as follows: AM = (NODES, DOMAINS, NETS, LINKS, PORTS, port, communication, status, event) (1) NODES = {set of static and dynamic information of nodes}= {NODES_S}  {NODES_O}  {NODES_A} where: NODES_S consists of system resource information of MO and this is static information such as Cpu, RAM, etc; NODES_O consists of information about local and communication operations such as processes; NODES_A consists of error or abnormal information of hardware and software components such as I/O errors, overload; NODES_O and NODES_A are dynamic information.DOMAINS = {set of domain information such as name,...} NETS ={set of network information such as IP, network,...} LINKS = {set of link information between nodes} PORTS= {set of port: internal and external port} port is a function that identify communication ports in NODES: local ports (internal) and external ports (send/receive to nodes not in NODES), port(NODES)  PORTS

Fig. 3 .
Fig. 3.The architecture model of a node With this architecture model, we can determine the related information of node such as resource information, operations, status, etc base on elements of AM.For example, give an architecture model AM of running node as Fig. 3 without communication operations, and then AM can be expressed as follows: AM=({n}, {d}, {net}, {L},{p 1 , p 2 }, port, communication, status, event) Where {n}={system information such as device name, CPU,...}  {running processes, I/O operations,...}  {error status,...}; port={internal ports: p 1 , p 2 }; communication={no communication with others}; status={up};

Fig. 7 .
Fig. 7.The monitoring implementation of ME_MO In order to observe and collect the architecture of MO in DS, we use protocols, APIs and built-in tools of operating system.The popular protocols are used in management network to monitor status or traffic of MO such as ICMP (Internet Control Message Protocol) [12,19], SNMP (Simple Network Management Protocol) [4,19].The APIs and tools are used to observe and collect system information, operations as well as communication ports of components in MO such as the Window API, Linux API, libraries, The modeling for monitoring solution bases on four levels such as MO, network, domain and global DS which are suitable with point of view the domain-based management, this hierarchical monitoring architecture have advantages to develop some distributed algorithms in levels of DS management in which the level MO focus on observing and collecting the architecture information of MO, level ME_MS, ME_MD and ME_DS are responsible for synthesizing and processing the monitoring information.

Fig. 8 .
Fig. 8. Basic monitoring process for architecture of DS

Fig. 10 .
Fig. 10.MCDS for the monitored object in VMSC3 model, network model, domain and global DS model.The architecture model for DS can be expressed as follows: Give a network consists of k monitored objects {MO 1 , MO 2 ,...,MO k } and set of {AM_MO 1 , AM_MO 2 ,..., AM_MO k } is architecture model of these objects.
 The architecture model of MO (AM_MO): AM_MO describe architecture information of MO and is expressed as follows: AM_MO = (NODES MO , DOMAIN MO , NETS MO , LINKS MO , PORTS MO , port, communication, status, event) (3)  The architecture model of a network (AM_MS):  The architecture model of DS (AM_DS): As DS is a set of n domains {AM_MD 1 ,…, AM_MD n }, so AM_DS is a composition model of AM_MDs respectively: www.ijacsa.thesai.org Because most of the built-in tools in M GM monitor DS by using discrete tools (OS or device vendors utilities), we have to type (or select) one or more commands respectively.M our monitor based on MEs and MEs communicate to each other by the message passing.So, monitoring time with M our will take less than M GM .In order to evaluate for monitoring time, basic monitoring time for object O i between M GM and M our are expressed as follows: t GM (O i ) and t our (O i ) are monitoring time of M GM and M our for object O i ; t mes (O i ), t Remote (O i ), t T (C i ), t R_P (C i ) and t' R_P (C i ): time for clicking function and monitoring message to object O i , remote to object O i , typing (or selecting) command C i , running-presentation results of command C i respectively.Suppose that t R_P (C i )=t' R_P (C i ) for same monitoring function in M GM and M our with the same O i .The experimental results are implemented in VMSC3, in which nodes work on MS windows environment.Result consists of some cases as follows:The hierarchical architecture model of monitored objects in DS and experimental result show that our proposed model is feasible and will overcome the disadvantages of specific builtin tools in monitoring hierarchical architecture of DS, as well as actively support administrators in managing DS in according to multi-level such as object level, network, domain and global DS level.Some actively results of above proposed model are presented in TableIII.

TABLE III .
SOME RESULTS BETWEEN BUILT-IN TOOLS (GM) AND MCDS