HA-OSCAR …attaining technological singularity(é) a step at a time

Its been a year since last winter quarter and once again HA-OSCAR (codenamed: Singularity) is on my projects ‘to do’ list this christmas. Last year, I removed all OSCAR dependencies from HAOSCAR making it a truly standalone high availability solution. This year, conclusion of my google summer of code project for OSCAR I gained a great deal of knowledge in the design of practical and highly modularized software systems that support third party extensibility. With this knowledge, it dawned on me that while HA-OSCAR was a first-rate concept, the way its concept was executed left much to be desired. In its current state, it was inflexible, static and only supported one (1) linux distribution.

With these flaws hampering any chance of wide adoption, I proposed a complete redesign and rewrite of its software. I am happy to say that its redesign was complete on the 30th of October and coding commenced in november. Currently, we are close to achieving milestone one. In this new redesign, a lot of non-standard HA components were replaced most notably ping with Linux-HA’s heartbeat. Some other additions include the inclusion of distributed replication block device (DRBD), Mon-it,  Kickstart & SiS suite and several abstracted components. I also hope to attain full compatibility in this new release with a number of linux distributions (CentOS, Rhel, Fedora, OpenSUSE, Debian, & Ubuntu).

As project lead and chief software architect of this new solution, I enforce strict coding rules to avoid the monstrosity the last HA solution had become. Python’s structure and concept of classes/modules plays a huge role in achieving this. In my next HA post I will discuss the four (4) core subsystems that make up the newly revised HA-OSCAR.

Below is the most recent project roadmap and goals of the new HA-OSCAR:

High Availability (HA) Computing has long been played a critical role in industry mission critical applications. On the other hand, High Performance Computing (HPC) has equally been a significant enabler to the R&D community for their scientific discoveries. With combination of HA and HPC, together will clearly lead to even more benefits to both industry, academic and research entities.

The goal of the HA Project is to create a flexible yet leading edge solution which seeks to provide a combined power of High Availability and Performance computing solution. It should enhance any computing infrastructure such as Webservers, and Clusters by providing the much needed redundancy for mission critical grade applications. To achieve high availability, component redundancy is adopted to eliminate single point of failures. Our HA-project incorporates a self healing mechanism, failure detection, automatic synchronization, failover and fail-back.

Roadmap for HA-OSCAR

…Valete Google

Well, echoing the posts of my fellow gsocers; the google summer of code has ended and my plans for the future might not be as murky as I initially expected. Its been 3 months since we were required to turn in our code samples, and ever since I have moved on to 3 new exciting projects. I will provide more in-depth information about my new engagements in my next blog post. For now here are some information and resources relating to my google summer of code project:

My Google SoC project was to develop and integrate into OSCAR a monitoring framework that was designed by OSCAR’s Oak Ridge National Lab team. The framework would be used by other developers who wish to integrate some form of redundancy in their applications by monitoring the state of various services. Detailed information on how to develop monitoring plugins based on the framework and testing this framework with an exisiting OSCAR installation can be found in the pdf version of this README located in the google repository.

Links:

Google Code Repository

OSCAR

Entrails of the OSCAR Monitoring Framework!

(GSoC update)

After about 3 weeks of coding I am elated to announce the completion of the gathering subsystem in the OSCAR Monitoring framework. The first plugin that made use of the framework was written and successfully updated the monitoring database with data. This plugin makes use of the Round Robin Database created and managed by ganglia

For those not familiar with my project here is a description of the goals of the project and here are the milestones achieved so far:

1. Definition of Schemas and familiarity with the OSCAR code base.

2. Successful implementation of the framework core(Interface to OSCAR Database).

3.  Extension of the core framework to the gathering subsystem.

4. Creation of a ‘oscargather’ (the gathering subsystem initiator).

5. Successful development and testing of an OSCAR component plugin.

With these goals attained so far, I can now being development on the next subsystem “The Actor”. This will supposedly take the monitoring data present in the OSCAR database and interact with the other OSCAR components thereby allowing for a more intelligent and dynamic cluster administration resource!

An Anatomy of the Gathering Subsystem

The monitoring framework is divided into 3 distinct parts:

1. Gathering Subsystem

2. Action Subsystem

3. Driver

This blog post describes the rationale behind the code for the gathering subsystem.

How does it work?

Any OSCAR component which wants to be monitored or can generate monitoring data such as heartbeat, IPMI, PBS_MOM e.t.c provides an xml configuration file which specifies certain properties of that component. This xml file will be located in the Gather subdirectory of the monitoring framework. A visual representation of the directory structure is thus:

When the gathering subsystem is initialized it parses all the “config.xml” files which contains all the configuration data for each component. The parsing is done with the help of the parser. The structure of the xml file will be discussed later.

The “Gather.pm” receives from the parser a hash of values, from this information, it is able to create an Oscar_Monitor table in the OSCAR database with column for each component to be monitored. The naming scheme for these columns will also be discussed later . With the Oscar_Monitor initialized, any component can then call the update method with an xml file and expect its data to be made available through the OSCAR database.

In the process of implementing the proposed framework, I saw that my implementation could not handle monitoring data from simple commands. For example, if I wanted to monitor the disk space available on the compute nodes I should be able to write a script to check this and some how put this information in the monitoring database. The task of putting the information in the Oscar_Monitor table is trivial but finding a way to continuously poll this script after a certain time interval would require the script writer to incorporate features that would ensure the script runs indefinitely so as to update its monitoring information available in the Oscar_Monitor table. Monitoring 10 different components with the C3 suite of commands will require 10 different scripts running indefinitely; eventually taking its toll on the processor. It was because of this problem that I created the oscargather daemon which is independent of the monitoring framework but has the ability to connect to the gathering subsystem to determine which components need to be polled if any and call its script every n seconds which will be specified in the components configuration files.

Starting or stopping the oscargather daemon is done with a simple “service oscargatherd start/stop”. This means all the 10 components which needed to be polled can be terminated or started cleanly by simply using this since each is just a thread of execution in this daemon. The oscargather also plays another vital role; it allows components(polled and non-polled) to be added on the fly by simply restarting the service. This is because restarting the service causes the monitoring framework to reinitialize itself therefore taking into account the new components added.

Structure of the “config.xml”

<?xml version=’1.0′?>

<monitorconfig>

<component> string </component>

<function> string </function>

<polled>Boolean </polled>

<script> string</script>

<time> integer </time>

</monitorconfig>

All components need to contain a config.xml file in the Gather directory.

Component: This is used to specify what component will generate the monitoring data for example you can have “Heartbeat”, “C3”, “IPMI”…

Function: This is a short description of the nature of the data to be generated for example nodestatus (for heartbeat), or diskfree (for C3). This is required because one component can generate monitoring data for 2 different purposes for example you can have C3_diskfree (cexec df –h) and C3_memoryfree (cexec free). In this case C3 is used in 2 different contexts. The naming scheme for each column in the Oscar Monitoring table is <component>_<function>. So for each component, its name and function separated by an underscore is created in the monitoring tables.

Polled: If a component wishes to be polled by the gathering daemon it should set this value to true.

Script: All polled components must specify a script name in this tag.

Time: This refers to the time interval for the polling.

Structure of the “data.xml”

<monitordata>

<column>C3_Diskfree</column>

<data>

<hostname>oscarnode2.xperiment</hostname>

<nodedata>diskfree 120GB</nodedata>

</data>

<data>

<hostname>oscarnode1.xperiment</hostname>

<nodedata>diskfree 140GB</nodedata>

</data>

<data>

<hostname>oscarnode3.xperiment</hostname>

<nodedata>diskfree 10GB</nodedata>

</data>

</monitordata>

Column: Specifies what column in the table to update.

Data: Each of the data tags contains two elements the hostname and nodedata. Hostname is simply the full host name of each compute node. The nodedata tag on the other hand is an anonymous data type meaning it is capable of storing data of any type. With this any generic data can be entered into the monitor table.

A full diagram of the whole system (without an implemented Action subsystem)

Advent of OSCAR Monitoring Framework

Benefits to OSCAR:
OSCAR is no doubt very popular as a cluster manager in HPC environments. This comes as no surprise when its high configurability and flexibility with regards to supported linux distributions is taken into account. Alas, installing a cluster resource manager is only the first step in managing a cluster; initiatives have to be employed to ensure that individual nodes in the cluster are somewhat resilient to failures. Incorporating a framework that allows monitoring of services running on the various nodes and storage of the information gathered using the OSCAR database will ensure a more robust OSCAR.

Project Synopsis:
Currently OSCAR can install a cluster, perform managerial tasks such as addition/deletion of nodes and also monitor the status of the cluster with ganglia or nagios. HA-OSCAR, an extension of OSCAR introduces redundancy at the head-node level by duplicating the primary head-node and based on predefined policies carries out specific actions to guarantee availability of this head-node. OSCAR cannot monitor the states of services concurrently running on all compute nodes such as lam , pbs_mom and take predefined actions in the case of failures. I propose the design, implementation and integration of a universal framework that allows gathering and storing information about the health of the various clustering services.

Project Details:
I will implement and integrate into OSCAR a simple universal framework that will be responsible for gathering information regarding the health of the cluster and storing this information in the OSCAR Database. Such an implementation will ensure that should a fault/error arise from any monitored service, there will be a mechanism to report this failure and based on user defined policies, actions can be taken to mitigate the effect of the failed service. The actual design of such a universal framework will be done in conjunction with my mentor while the implementation and integration into OSCAR core will be done by me.

Failure as a sure pathway to success…

Its been a month since I received the morale diminishing news from Stanford, what have I done so far?
First I was accepted into the google summer of code program to implement a high level generic monitoring
framework also we successfully illustrated a prototype of our robot in the robotics competition.
If the phrase a journey of “a thousand miles begins with one step” is any consolation for the tumultuous nature life’s pathways to success are I can justly say thats one big step I’ve taken.

Lesson learnt from my most recent experiences seem to reinstate the notion that giving up on goals, aspirations or dreams in time of failure can only be an indication of weakness but rather cultivating a persistent approach no matter how grand or seemly impossible it might initially be can unlock unseen opportunities .

In my next post, I will discuss my work so far on the google project!

The best damn SVN client for macs

 

Sorry if you expected this page to be full of praises for the apple operating system. Without digressing I will state what is obvious to any rational, open minded person that has not being influenced by any “fan boyism” .

The best damn svn client for mac is SyncroSVNClient period!

The only drawback is its non-free nature. After looking at most available svn clients for mac I came to this conclusion. Ironically, macs are renowned for their ease of use and programmer friendly environment yet the only svn client worth mentioning is one that does not include half the functionality of tortise svn in windows. I find this disturbing.

The interesting prospects spring always brings….

I place my fingers on my keyboard and try to put my otherwise chaotic thoughts on screen but nothing is forthcoming. What could the matter be? Am I suffering a case of writers block? or could it be that my thoughts are meaningless and their entropic state is simply a reflection of my dreams; unpredictable and disorganized.

I have a simple one word answer to all the various questions posed above…. ‘EXCITEMENT’ . Yes excitement, within the next 1.5 months I will have made my first meaningful robot, a robot whose intelligence can be applied in real life situations, an industrial robot. But you see, that is only half of the reason for my excited state, I also get to introduce some more advanced features into HAOSCAR which can only make it better at what it does (providing high availability). With these two projects occurring simultaneously I have every reason to be excited. Enough about me…lets get into the nitty gritty of the details.

In the past I could only dream of one day taking part in a robot competition but this quarter this dream will be realized. This year the competition will be a marathon of three (3) events; maze solving, sumo wrestling and finally best design competitions. The robot we (my team of two) will be building will not be taking part in any of these events instead we volunteered to build a robot that could be used as an industrial robot. This robot will be required to locate cans of particular colors and transport them to different pre-determined locations depending on the color of the can. To add to its complexity the robot will be required to navigate around barbie doll obstacles in its path . A link to the full description is here

In the case of HA-OSCAR, I was charged with the task of introducing a new level of resiliency by integrating a distributed replicated block device (DRBD). Modifying HA-OSCAR to install and configure drbd and also modifying drbd to adhere to the policies of HA-OSCAR will no doubt be challenging.

With all these, it is evident that I have yet another very interesting quarter ahead of me therefore I have every reason to be excited.

In my next blog post I will describe in detail the algorithm we will use to solve the industrial robot problem and also update information about the progress of HA-OSCAR.

Its that time again….Robots, Networks and Clusters make one happy man.

I don’t know why I am making this post at 3:00am when I should be sleeping..well, the quarter is drawing to an end and all looking back I can describe it with one word “Edutainment” (quotes needed). “Edu” is from the word education meaning this quarter has been highly informative. I learnt quite a lot of things ranging from what makes the internet work(routing) in networking to what makes a super computer super in high performance computing. The “tainment” part of course refers to how interesting and mostly fun it has been for me. I wrote programs showing how routing algorithms such as RIP and OSPF work, programs the are highly scalable running on our 32 node cluster and even programs that make use of highly parallelized and specialized hardware clusters like the Nvidia General Purpose GPU and Playstation 3.

Is this the most fun I have ever had? The answer will be no. I think next quarter( Spring) holds even much more fun. I actually get to take a class I have been dreaming about for years. The advanced computer architecture. This will be my formal introduction to the field of robotics. We will build robots to solve very complex maze problems with basic sensors such as Infra red, sonar, image processing sensors and hopefully the cheapest laser range scanners we can find. I can already see the applications of this quarters’ classes. What can clusters, networking and robotics give you. The answer is one word “Intelligence”. We can build a robot that connects to the school cluster via wifi to perform image processing tasks. Since this is compute intensive and network latency also plays a major role, the image processing aspect of the robot will only be used for long range perception. We can incorporate the sonar and infra-red sensors for “immediate sensing”. I go to sleep this night full of excitement and anticipation

…one thing is certain, after this academic year, I will have carved my own niche in computer science, the language of gods.

 

UPDATE: Instead of using a traditional cluster with network latency issues, we can explore some GP GPU image processing since it performs exceedingly well in huge matrix computations…^_^

Reviewing HA-OSCAR…

During the course of taking the HPC class in my school, we were required to write different research papers depending on what research project you were assigned to do. Since I was assigned the task of removing several OSCAR dependencies in HA-OSCAR and re-integrating these into its core, making it a complete HA solution for any cluster or server platform , it was only natural to have been assigned the task of review it against some other prominent HA solutions. A link to the paper and power point presentation is here.