The Microsoft Chapter

After joining Microsoft, I think it is time to split my blog posts. So from now on all Microsoft related posts will be published on my new blog at MSDN Blogs.

I will keep posting on this Blogger site about the open source projects. Stay tuned.

#SNMP Design: Specification Document Draft Published

This is an invitation to everyone interested in #SNMP.

Like other open source projects I maintain in the past, #SNMP lacks of documents. Although I have a lot of posts here about it, I think it is not yet enough for users to understand the bits. Therefore when I received a mail from a student, I found it is hard for me to help out.

Hi,

I am working on the SharpSNMP project as my graduate course work, to build more Unit test cases on the SNMPlibrary. I see that there are already built test cases for the basic SNMP functions like the MIB verification and variable verification. But, it is very difficult to understand the flow of the code base. It would be great if you could supply us with any documents on the code base that you have.

Thanks,



Starting from yesterday, a new document is under work to publish the information this student, as well as other users, may be interested in. It is a specification file in Google Docs here .


Please provide me comments so that I can improve it weekend by weekend. :)

Important Notice

OK, some of you may notice that I only appear on my blog at weekends these days and am no longer active on the open source projects. I need time to adjust. So please forgive me if I reply to your comments or questions late. See you next weekend. Stay tuned.

#SNMP Design: One API Design Flaw

When I reviewed the message classes, I suddenly realized that there is a design flaw. Do you notice?

Let's forget about existing code, and think about this case. You want to query system description from two agents in the network, so how to write some code? Probably this is the simplest code,

    Message message = new Message(/* arguments */);
    message.SendTo(agent1);
    message.SendTo(agent2);

Therefore, when you create the message instance the IP address should not be specified. Yes, the current design fails to support this simple use case, which is obviously a design flaw. But why I did not realize this flaw earlier? Why the early design is like this? That is because TrapV1Message was the first message class, and to create its instance an IP address must be specified. Now I fully understand why people think SNMP v1 TRAP is a bad thing, it does not fit in to the whole image.

At this moment, I don't want to change the existing code because TwinTower is almost ready. Thus, this change will be introduced for CrossRoad. OK, next time I may provide backward compatibility (current APIs will still be available but marked as obsolete while new APIs are introduced side by side. Time after time, I will remove obsolete code so you have time to prepare the migration).

Stay tuned.

#SNMP Design: Home-Made Logo

OK, I have been thinking of a logo for #SNMP for a long time. Now I posted a draft version here, and please send me your comment. Thanks. :)

#SNMP Design: Discussion on Manager.Walk

OK, according to Producing Open Source Software, there should be no private conversation under the surface, so I am going to publish a few conversations between me and Steve so you can be updated.

This time, let's start with a talk about Manager.Walk.

Steve: Crowe's libary had a ManagerSubTree. An older version of an app uses Crowe's library. I was tasked to switch that app to use our stuff, and I don't know how you want to deal with this. pretty much they used it in a way to see all of the print jobs in the table, and then goto the last one and return information on that index. I know we do have Walk, which returns a one dimentional list of answers, but I guess I need more help in understanding how that works and stuff.

Lex: In fact, when I started to write table related code for #SNMP, I did not refer to Mr. Crowe's code any more, so I really don't know how ManagerSubTree works (I will investigate it later). Therefore, I cannot easily say if #SNMP can do the same or not. 

Lex: Hi now I think I can answer the question about "ManagerSubTree" class in Mr. Crowe's SNMP library. Yes, although it appears like a class, the function it supports is similar to Manager.Walk in #SNMP. Manager.Walk returns a list of objects (depends on WalkMode argument), and ManagerSubTree provides the same list from its indexer or GetEnumerator() method (and it only walks in the subtree). 

OK, now let's take a look at another key difference. In ManagerSubTree, there is a fix tool to fix something against a non-compliant agent. #SNMP library does not yet have similar code and I don't know if it is necessary.

Hope this also helps you understand the relationship between Mr. Crowe's SNMP library and #SNMP.

Stay tuned.

Open Source "Must Read" List From Me

In my opinion, there is few great books about open source out there published in China. And I was lucky enough to find out two free and great books during these years.

Open Sources: Voices from the Open Source Revolution 
I spent a long time to find my first job opportunity when I was at college, so one of those days in late 2006 I happened to be near one of the biggest book stores in city of Wuhan. Therefore, I paid it a visit and found a copy of Open Sources on a shelf. 

I thought it would be hard to read those words from technique geeks, but I was totally wrong. The book is so easy to understand and it was so much fun. You may take a look at it here for free. O'REILLY is such an amazing publishing company that it provides many great books for free.

But this great book is full of philosophy and history, so as a FOSS developer you get no idea how to play with open source yourself.

Producing Open Source Software: How to Run a Successful Free Software Project

This must be the best guide to prepare yourself for open source projects. Once you understand the book, you can decide how to participate in the movement. I just found this book so I am not yet finished. Hope that I can finish soon. The greatness of this book is that Karl the author shows you the best practice so I can see what I did wrong in the past, and what to do right in the future. And, wow, this book is also published by O'REILLY .

I did not have chance to read other open source specific books, so if you have some recommendations, please leave a comment. Thanks.

#SNMP Design: Incomplete Agent Demo

[Update: This post is obsolete. For #SNMP 6+, please read this new post http://www.lextm.com/2010/11/honeycell-drops-snmp-pipeline-and-our.html]


I started to increase agent side support in #SNMP, but soon I found that I could not do that fast. It is really a big topic. But I owe everyone who voted on the poll a detailed post, so here comes it.

I am now guiding you to create a WinForms application that can simply handle one SNMP request as an SNMP agent. You may follow the steps to see how many pieces are still missing in the map. By the way, if you are interested in this field and want to contribute, drop me a comment so I can contact you.

Project Initiated

Create a new WinForms application called AgentDemo in Visual Studio or SharpDevelop.

Then add #SNMP library project into this solution. 

Now you can add a project reference in AgentDemo to SharpSnmpLib.

Basic GUI Setup

If you take a look at the Toolbox at this moment, you can see three new components,

Please drag the Agent component onto your Form.

Also you need to add two buttons like this,

So the simplest UI is done.

Simplest Code

So now let's see how to bind event handlers. The Start button must do this,

        private void btnStart_Click(object sender, EventArgs e)

        {

            agent1.Start(161);

        }

while the Stop button does this,

        private void btnStop_Click(object sender, EventArgs e)

        {

            agent1.Stop();

        }

Note: If you don't explicitly specify the port number in Agent.Start, 161 will be used by default.

And the hardest part is the agent related. Thus, I handle the GET requests here for demo.

This is the simplest GET request handler I can think of,

        private void agent1_GetRequestReceived(object sender, Lextm.SharpSnmpLib.GetRequestReceivedEventArgs e)

        {

            GetRequestMessage message = e.Request;

            MessageBox.Show("A request is received from " + e.Sender.Address);

        }

Why Not Response?

Now I will show you how to response to a simple request. Modify GetRequestReceived handler like this,

        private void agent1_GetRequestReceived(object sender, Lextm.SharpSnmpLib.GetRequestReceivedEventArgs e)

        {

            GetRequestMessage message = e.Request;

            // you may validate message version number and/or community name here.

            if (message.Variables.Count != 1)

            {

                return;

            }

            if (message.Variables[0].Id != sysDescr)

            {

                return;

            }

            GetResponseMessage response = new GetResponseMessage(message.SequenceNumber, message.Version, e.Sender.Address, message.Community, new List() { new Variable(sysDescr, new OctetString("Test Description")) });

            response.Send(e.Sender.Port);

        }

        private static ObjectIdentifier sysDescr = new ObjectIdentifier("1.3.6.1.2.1.1.1.0");

In this way, AgentDemo.exe should be able to reply simple requests about its system description.

You can also send out INFORM or TRAP messages by calling Agent.Send* static methods to SNMP managers.

Note: You may find this code cannot compile even if latest source code is used. Sorry, you have to add the following code to GetRequestMessage.cs,

        ///

        /// Version.

        ///

        public VersionCode Version

        {

            get { return _version; }

        }

        

        ///

        /// Sequence number.

        ///

        public int SequenceNumber

        {

            get { return _sequenceNumber; }

        }

        ///

        /// Community name.

        ///

        public OctetString Community

        {

            get { return _community; }

        }

The Problems

Although this small demo works, I cannot say #SNMP can be used to develop agents. There are serious problems,

• There is no built-in support to store managed objects in #SNMP, so you have to store them yourself.
• It is not easy to query these objects and generate proper response messages. You have to write a lot of code to realize the queries and message generation.
• If you need to manage a lot of objects, there is no support to generate code from MIB documents for you like other commercial libraries.

It takes us a long way to get here (wow, six months has passed), but more work is ahead.

References

I have blogged about agent side support in the past, and here are the links,

The Poll

Send Trap

Poll Result

Basic Design

#SNMP Design: Changes from Steve, and Latest News

Steve have just checked in a few changes on the browser side,

• Table View. When you navigate to a table node in the Tree, you can right click and use Walk menu item to launch the Table View dialogue.
• Save in Output panel. Now you can save logs in this panel to an external file. This is convenient for debugging but also useful if you want to back up the logs.

The SET dialogue is the last piece to enhance for TwinTower on Steve's list, so we can expect RC3 to be released soon.

If you monitor the Discussion board, you may find out new bug reports like this one. OK, this one really reveals that the old versions must be retired as soon as possible. After checking the Issue Tracker, I suddenly realized that I have done so many during August and September. Therefore, now Unicorn Refresh is no longer the default release. 

Please try out TwinTower RC2 from now on. Although both Steve and I checked in changes later, the RC 2 interfaces are not modified.

Guide on Microsoft Hyper-V Server 2008

Because Microsoft Hyper-V Server 2008 is a free hyper-visor, you may find it hard to search for related materials. I felt the same way but luckily I came across this post by Ben . It is so nice that he provided a step by step guide. Unfortunately I did not have a second computer which is required to set up the management UI, so my experiment could not be accomplished lately.

#SNMP Design: Take A Break

Now my 9-day-long vacation is going to end. It is really nice vacation because,

• I have chances to visit big stores here to purchase shirts, trousers, and shoes,
  
• I can enjoy 10-hour-long sound sleep every day without any interrupt,
• ...

But the most important thing is that I can put into practice a lot of ideas for #SNMP. This is a short overview of what I have coded these days,

1. Command line MIB compiler prototype created,
2. GUI MIB prototype created,
3. 1000+ MIB test passed,

OK. The list is short, but it is not easy to accomplish the items. I never expected I could make them weeks ago, but now dreams come true. Hope you enjoy these prototypes and provide your suggestions.

I feel bit of nervous because I will be on-board for my new job at Microsoft next Monday (October 6th). Therefore, I have to take a break from #SNMP and start to prepare.

#SNMP Design: 1000+ Documents for DockPanel Suite

Now #SNMP is not the only one who suffers the performance issue. I have just found DockPanel Suite has a similar issue. For example, I created a simple DockContent derivative that simply contains a RichTextBox to host the MIB documents. Then I followed the usual way to design the main window and let it open 1000+ documents. Wow, it froze there for a long time.

Luckily I had all the time in the world to wait till all documents were opened (about 195,000K memory is consumed). And now you can see how the compiler GUI prototype looks like,

Do you like it?

(Updated: OK, just found out that both Visual C# Express and SharpDevelop 3 Beta froze and died in the same case. Wow. Maybe opening 1000+ files at a time is a crime so nobody assumes that I would ever commit it.

I have just found a workaround. Why not force the users to open <100 files at time? Remember it is still a prototype? In this way even if they want to open 1000+ files, they can try to add all files in five rounds.)

#SNMP Design: The MIB Compiler GUI

If you played with commercial SNMP products for developers, you may notice how they were designed. For example, SilverCreek combines the compiler with the browser while MGSoft splits them into separate executables. Therefore, I think it is time to decide how to design #SNMP Browser and Compiler.

Plan A 

In this plan, the browser has less features than it is now. The feature set is,

• Loads *.module files from the module inventory (Yes, instead of MIB inventory).
• Display a object tree.
• Support SNMP operations such as GET, SET, GETNEXT and so on.
• Support TRAP and INFORM.

Then these features will be added in the compiler,

• Compile new MIB documents to *.module files and add to inventory.
• Delete module files from the module inventory.

Plan B

In this plan, the compiler will be part of the browser. So all features described in Plan A will be available in the browser. 

However, this means there will be a few more panels and menu items added to the browser which may be an issue for collaberation. Do you know it is hard to merge Form and UserControl in WinForms applications?

Current Approach

I was interested in Plan B at first, and that is why you can add and delete MIB documents from the current MIB browser. 

But Plan A sounds better, because Steve is willing to maintain and improve the browser. I don't want him or myself to waste time on merging changes from both of us simply because of WinForms.

Therefore, now I am going to create a new project file in #SNMP Suite, named #SNMP MIB Compiler following Plan A. This GUI compiler will also use DockPanel Suite to host panels. 

Although it starts as a separate project, I think it is possible to integrate the panels into the browser if necessary in the future. Yes, that makes Plan B always available as a back up.

At last, I want to state that I am always open to suggestions. At least, the compiler is going to be official released in CrossRoad, so there is plenty of time to adjust it.

#SNMP Design: Oh, 63 Seconds!

1465474 milliseconds is a long time to load 1000+ MIB documents into the current browser in #SNMP (see this post for details). Do you think 24 minutes is a reasonable amount of time? I feel it too long. 


When I wrote this post about the new design for CrossRoad, I understood that it could boost the performance. So the Turbo comes now. I am going to check in the latest changes in a few minutes, which is able to unbelievably do the same thing in 63410 milliseconds (63 seconds). 23x is a really nice result for a prototype, considering I only made use of existing Compiler and ObjectTree implementation.


It is sad that the browser cannot enjoy this turbo engine until I integrate it with the current browser. This integration may take a few weeks because Steve not yet finish the merging. As a result, I have plenty of time to improve the prototype lately.


In the next few months, I will refine this prototype until it is ready for CrossRoad release. But from now on, you can also play with it to see what is the magic. Stay tuned. 


(Updated: One night later it only takes about 20-30 seconds to load 1000+ MIB documents. What I have done is simply move a few methods from Assembler to ObjectTree. It is hard to explain why this leads to another 2x boost.)

#SNMP Design: Oh, Dear Compiler

I cannot believe that a prototype of the proposed MIB compiler is now working on my computer. And during the process of developing it, I merely added a few new classes and methods,  but no API change was made since Change Set. Nice, I think Steve is happy to hear this.

All Net-SNMP MIB files was compiled to *.module files a few minutes ago, and now the 1000+ MIB documents are being processed. Although the compiling still takes a long time, I expect that the browser can load 1000+ *.module files in a few seconds. If later I can prove this, then I think I have already solved the performance issue and go beyond my original design of the browser and the library.

In the old design,

• Net-SNMP MIB documents are bundled in the library and parsed every time a ObjectRegistry is created. 
• The browser accepts MIB documents directly, and parses them every time to form a tree.
  

If the compiler works as expected, then these changes can happen immediately,

• Only *.module files are needed by the library to create a basic tree.
• The browser can use *.module files by default, and compile MIB documents to *.module files whenever a new document is added.

Isn't that wonder? I am going to prepare a performance report of this compiler prototype soon. Also, I hope I can create a GUI for this command line compiler, too. Luckily if all these new stuffs do not break the existing interfaces the browser relies on, I can make sure they won't delay TwinTower.

#SNMP Design: Compiler Design Proposal for CrossRoad

You may notice the current official release of #SNMP is still UnicornHorn Refresh (1.1) which is available since August. Now TwinTower (1.5) RC2 is out, and the final release will be published weeks later. So it is time to define detailed baselines for both TwinTower and CrossRoad (2.0) that match the situation.

In this post , I ready stated that no new features (even the performance tuning) will be added to TwinTower except bug fixes. So the following features will definitely be delayed to CrossRoad,

1. MIB compiler prototype.
2. MIB browser performance improvement for 1000+ MIB documents.
3. SNMP v2c missing pieces.

So in this post I am going to provide some ideas about the MIB compiler design.

You already notice I start to call the MIB handler in #SNMP "a compiler". This is not yet a true compiler, but the structure is similar. I have divided the whole process into two phases, parsing and assembling. The current structure is not yet completed, so the following problems remain,

• The parser does not generate enough information for the assembler. For example, when the assembler receives an IEntity object, only its parent name, name, and value are available, so a recursive handler must be there to assemble the object into the tree. 
• The assembler cannot use an efficient way to assemble the entities. Take a look at ObjectTree.Refresh, and you will see how complex it is. This is a direct result of the first item.
  
• The parser cannot report progress, errors, and warnings.
• It is even worse if there are complex OID definitions defined in the MIB documents. The assembler consumes more time to locate their position in the tree, and assemble them. In some cases place holders must be generated by the assembler to make sure the node can be attached correctly. 

So the new design contains a few new ideas to simplify the entire process.

Parser (Front-End)

In order to make the assembler easier, the parser must generate as much information as possible. For example, if the complete OID of an object can be generated by the parser, then the assembler can use a much more efficient way to assemble the object by calling ObjectTree.Find(uint[] numerical).

However, this means the object location must be determined by the parser instead of the assembler. So during compilation, the parser must retrieve the tree to know where to place the new nodes, and create place holders if necessary.

Because the parser can be separated from the browser completely in the future (if the assembler in browser only reads in module files), the time consumes by the parser will no longer affect the browser performance. This separation can significantly simplify the browser. 

Module File Format

It is possible to store those objects into text files just like a real compiler. For example, define a .module file format to contain the objects,

1. The file name is the same as the module name.
2. Every line is an object.
3. The object is stored in a comma separated way. ;;;(;)

Then whenever a MIB document is imported, such module files are generated by the parser. It is much easier for the assembler to read in these module files and build up the tree from scratch.

Assembler (Back-End)

Now the assembler simply read in module files, and make sure the nodes are generated correctly. This "thin" assembler can work much faster.

I may check in latest research work into the repository periodically, but surely the final work is going to be included in CrossRoad.

#SNMP Design: The First Performance Analysis

I never take a look at performance counters until it is time. And now it is time for #SNMP. According to my latest analysis, it takes about 24 minutes (1465474-milliseconds) to load 1000+ MIB documents in the browser. This is horrible. So where is the bottleneck?

A further evaluation is now under way. Luckily I have split the compiler into two parts, front-end (parser) and back-end (assembler), so it is easier to time the operations separately.

Out of my expectation, the parser works fine. The time it takes to parse a MIB document is about tens of milliseconds (min: 0-ms, max: about 733-ms for POWERNET-MIB). So it is the assembler that has poor performance. It takes handreds or more of milliseconds (min: 0-ms, max: 57096-ms for VBRICK-BOX2-MIB) to assemble all entities of a MIB document to the tree.

Now I realize the issue and I start to explore it. Hope I can tune it soon.

What about the schedule? Although RC2 is already out, 1.5 release is not yet ready. Steve will work hard to merge all changes later to provide a stable browser. Then an evaluation report will be provided so we can discuss what are missing. If we think all basic functions are there, then 1.5 RC3 will be delivered immediately. One or two weeks after RC3, we can ship the final release of TwinTower. It takes much longer than expected, but we already finish more (such as SNMP v2c support, basic Agent component, and so on).

Stay tuned.

BTW, don't be panic about the performance issue. Not everyone has 1000+ MIB documents, isn't it? :) Up-to-now, I am the only one who suffers.

#SNMP Design: Another Parser Milestone, 1000+

On September 11, I received a mail from Robert Arnold that he can provide a bunch of MIB documents. So I asked him to package them to me. All those documents are taken from this web site. Robert wrote some scripts to grab documents from the pages, so I don't need to navigate and copy contents.

It was two weeks later that I finally had time to work on those documents. Therefore, a lot of bugs were found in the parser and fixes were located as well. When I checked in Change Set 16025 yesterday morning, the parser was significantly improved. But more bugs were found in the browser once so many documents were available.

So I spent almost one day on the browser. Several parts were refined,

MIB Inventory
Steve added a lot of code inside ObjectRegistry and ObjectTree to handle user imported MIB documents. But I think it is better to handle them in an individual unit. So MibInventory class is now added.

Compiler Units
The parser/compiler is now splitted into two logical units. The engine is now separated into a new unit called Compiler, which serves as the front-end (input: MIB documents, output: module entities). The back-end is still part of ObjectTree who assembles all module entities into an object tree (input: module entities, output: MIB object tree).

Recursive Assembling
A critical bug is found in the back-end that many entities are not assembled because of their complex dependency on other entities. This requires a recursive algorithm inside so most of these ignored entities can be assembled round by round.

Complex OID Definitions
In the past, it is rather hard to handle { iso 2 840 10036 }, { iso(1) org(3) dod(6) 1 }, and so on. But now both front-end and back-end can support those definitions correctly.

So now I am going to check in the latest updates. The following picture sets a new record,

1217 modules loaded in Module List panel, while 61 modules pending.