Software guru, Inc

Real World Solutions

The Guru's Blog

MSMQ: The Indispensable Windows Subsystem Most Developers Have Never Heard Of

Jan 14

Written by:
1/14/2013 10:44 PM  RssIcon

MSMQ: The Indispensable Windows Subsystem Most Developers Have Never Heard Of

MSMQ, Microsoft Message Queuing has been around since the days of NT 4.0 and Windows 95, way back to 1997.  Initially it was an add on product to windows, but starting with Windows 2000 it was included in the server versions of Windows and with Windows XP it was included in the desktop operating system.  It’s a Windows Feature that you can install/enable via the Windows Components part of Add/Remove Programs (or Add Features in W2K8).  Yet surprisingly few application developers know about it or how to use it to improve application performance.  Tonight we start with an introduction, later posts will talk about improving desktop application performance (everyone who blogs about MSMQ talks about server uses of MSMQ)

It’s one of the great secrets of high end data processing programming.  Before MSMQ people would pay a lot of money for MQ Series from IBM (now called Websphere MQ) or similar programs from other third parties.  (People still do, by the way).  However, thanks to Microsoft’s generosity, all Windows machines get it for free.

Traditionally it’s been used to facilitate communication between applications and servers running at different speeds and priorities ensuring no one machine was held up by processing on another.  It’s still used for that, of course, but there are a lot of other things you can do with it, in particular on the Desktop and on the Web.

So what is it?

Simply put, MSMQ is like an e-mail/FedEx for programs, which makes MQ Series like an e-mail/UPS for programs.  It’s (optionally) guaranteed delivery of messages between different applications running either on the same, or different machines.

Unlike real email, MSMQ (and Websphere MQ—implied from now on) is extremely fast, extremely reliable and extremely secure.  The API allows routines to send and receive messages back and forth via message queues.  Queues are stacks of messages that operate, by default, on a FIFO (First In, First Out) system.  Each routine that wants to talk to other routines has one or more queues associated with it, and there is often, but not always, a directory service that allows it to locate other routines (although this is generally dictated by the programming logic).

So, what are these messages?

Messages are binary blobs of data with a structured layout that the user/programmer doesn’t have to know about.  In the API they appear as a “Message” object.  The message object has a number of properties and methods, much like an object representation of an e-mail message.  Some of the most useful/obvious ones include:

·         Label

o   The Name/Subject of the message, user defined string

·         Body

o   The Body/Contents of the message.  It’s a binary blob, use the Body Type to determine the format of the message.   Traditionally the body is text or XML, but it can also be a binary stream.

·         Body Type

o   Tells the receiver how to interpret the body.  There are a number of predefined types and one can create custom types.

·         Correlation Id

o   Generally a user generated field to tie this message to some other message, for example to specify a conversation.

·         Priority

o   An integer between 0 and 7 indicating the priority of the message, the default is 3.  How priority is interpreted is up to the developers on the sending and receiving side.

·         Message Class

o   What type of message is it: Normal, Acknowledgement, Negative Acknowledgement, etc.   Most people only care about normal messages, but if you are tracing the delivery of your messages, you’ll want to deal with the others (think Delivery Receipts)

·         App-Specific

o   An application specific unsigned integer.

o   I use it to define application message types to aid in quickly processing messages.

·         Extension

o   One of my favorites, a user defined binary field, I think of it a binary TAG field.

·         Destination Queue

o   Where the message was sent to, or about to be sent to.  The “TO:” field, where the address is the pathname of the queue.

·         Response Queue

o   Optionally where any replies should be sent to.  The “REPLY-TO:” field.

·         Sender ID

o   The UID of the person or application sending the message.  Used for authentication purposes (there are also Authentication fields, Encryption fields etc.) 

§  If I have user desktop applications sending MSMQ messages, I’d have the UID of the user in this field to determine if they are authorized to request my receiving application do whatever it is the user has asked. 

§  This is different than permission on the queue itself, which can also be set.  I may want to allow a wide range of users to submit messages to this queue, I just may wish to limit which types of “requests” those messages might contain, so I check the Users Windows Id.

·         Delivery

o   This one gets overlooked, to many people’s dismay.

o   Set this property to MQMSG_DELIVERY_RECOVERABLE to ensure that when MSMQ gets restarted, the message stays in the queue.

o   The other option MQMSG_DELIVERY_EXPRESS creates non-persistent messages that go away when MSMQ restarts.  This is the default setting.  There are good reasons for this, but I won’t have time to go into them here.

 

What are Queues?

 

This was a lot harder to explain to people 15 years ago, but…

 

The best way to understand a queue is by analogy to something modern workers are all familiar with.  An e-mail Inbox.  Imagine if you will, that your job, or part of your job is responding to messages that come into your Inbox (or maybe a set of Inboxes/Public Folders)?  You have to at least look at every message and decide what action to take, if any, and what response you must provide, if any.  And as long as you are working you have to do this.

 

Now, imagine you are an application, a Message Queue is an inbox that you are responsible for monitoring.  You might have a main Inbox and you might also have a bunch of other Public Folders that receive email.  All of those are Message Queues and the application has to process any messages that come into the queues (folders/mailboxes) that are assigned to it.

 

That’s it.  That’s all there is to it.  Basically.  Yes you can get complicated with priorities, dead letter queues, journal queues, NDR’s etc.  But all of those have e-mail equivalents.

Queues are inboxes of messages, the API provides programs with a way to either Peek or Receive into the queue.  Receive is sort of the default option, it’s basically “give me the next thing in the queue, the oldest thing.”  When you receive, you pull the message out of the queue, and the message behind it moves to the top.  If you decide you don’t want to deal with it, you put it back into the queue, where it goes to the end of the line (unless you do something about it).

 

Peeking on the other hand is just taking a “peek” at the message without removing it.  You can basically peak your way through all the messages in the queue to get an idea of what’s in there.  There are various routines that build on this to give you lists sorted by priority, response queue, correlation-id etc. 

What properties do queues have?  Actually not that many, here are the major ones:

·         Label

o   A friendly name for the queue

·         Pathname

o   There are different formats for this but basically it’s going to be \

·         ADS_PATH

o   If you store the queue information in Active Directory Services, it’s the ADS path.

o   To be clear, it’s an LDAP path. LDAP://MyLDAPServer/CN=MyQueue,CN=msmq,CN=MyComputer,CN=Computers,DC=MyDomain,DC=MyCompany,DC=COM

·         Multicast Address

o   If you want to use multicast to address the queue, this is its address.

·         PathName_DNS

o   DNS pathname for the queue

·         Base Priority

o   The default priority level of the queue.

·         Journal

o   Specifies whether the queue is journaled.

o   If a queue is journaled, it has a journal queue, which is a queue that all “received” messages go to.  When you “Receive” a message, it vanishes from the queue.  If the queue is journaled, a copy of that removed message goes into the journal.

·         Type

o   A GUID that identifies what service is associated with the queue.  With a few exceptions, this is user generated, generate a new GUID that’s unique to your application for this purpose.

o   It’s used so applications can find queues to monitor via the directory service.

·         Quota

o   Maximum number of messages that can be in a queue, if you want to put a limit.

·         Transaction

o   Is the Queue Transactional.

o   Here’s the cool part:  Messages can be delivered transactionally to ensure deliver/respond to failure.  Use DTS between servers.

Those are the main ones, there are a few others for encryption and security.  However, on top of this there are some innate properties of queues that are determined at the system level.  In particular: Queues can be either public or private.  In general private queues are used within a single application and single machine and public queues are used for applications to talk with each other on the same or different machines.  This property defines some of the search criteria for queues when using the directory service.

How do you Access this stuff?

In .Net use the System.Messaging library, you can also use many other variants and in particular the COM objects. 

Queues can also have Triggers on them.  Triggers are actions defined at the system level that fire when messages arrive in a queue to which the trigger is attached.  I don’t particularly like these things.  Again, I’m old school.

My preference is to either have Windows Services that monitor queues, or a regular Windows Application that listens to queues.

At its most basic level, sending a message is a simple as creating a Message object, filling out its properties and then using its send method to send it to a queue whose name you happen to know.  If you don’t know the name of the queue you can search for it via the API in a number of different ways.  If you know the name of the server the queue is on, you can request a list of all public queues and look through them.  Otherwise you have to rely on a directory service such as DNS or Active Directory Services to find your queue.  I myself generally build my own custom search logic via name patterns and lookup records.  This is simply because I’m old fashioned.

Again, at its most basic, receiving a message is simply a matter of knowing the pathname of the queue your application wants to monitor, or looking it up, and then Opening that queue and either Peek into the queue, do a Synchronous Receive (you will lock until a message arrives or there is a time out) or do an asynchronous BeginReceive with an attached event handler.

In this last case when a message is received the event is raised and your handler gets the message as an argument and can then process it.  The trick is to remember to call BeginReceive again when you are done processing.

Of course, the other, unstated trick is to know if someone/something else is Receiving from the same queue.  No different than if you have two or more people answering the same email.

Some examples?

First we have a simple form that has a Start Button, a Stop Button and a RichTextBox.

Its purpose is to receive text messages from another user on the (presumably) the same machine [Change the computer names on each side to put them on different machines]

 

Imports System.Messaging

Imports System.Security.Principal

Public Class Form1

    Dim MSMQ As System.Messaging.MessageQueue

    Private Sub StartButton_Click(sender As Object, e As EventArgs) Handles StartButton.Click

        Dim MyName = My.User.Name

 

        Dim MyQueueName As String = My.Computer.Name + "\" + IIf(MyName.Contains("\"), MyName.Substring(MyName.IndexOf("\") + 1), MyName) 'Gets rid of the domain part of the username.

        If Not System.Messaging.MessageQueue.Exists(MyQueueName) Then

            System.Messaging.MessageQueue.Create(MyQueueName)

        End If

        MSMQ = New System.Messaging.MessageQueue(MyQueueName) 'Creates a new instance of this object and attaches to the queue of this name.  Does NOT create a new queue.

 

        'The next part is optional, but I want to specify what fields get pulled back automatically when looking at the queue

        MSMQ.MessageReadPropertyFilter.SenderId = True

        MSMQ.MessageReadPropertyFilter.AppSpecific = True

        MSMQ.MessageReadPropertyFilter.Label = True

        MSMQ.MessageReadPropertyFilter.ArrivedTime = True

        MSMQ.MessageReadPropertyFilter.SourceMachine = True

        MSMQ.MessageReadPropertyFilter.Body = True

        MSMQ.MessageReadPropertyFilter.Priority = True

        MSMQ.MessageReadPropertyFilter.CorrelationId = True

        MSMQ.MessageReadPropertyFilter.Extension = True

        MSMQ.SynchronizingObject = Me

        AddHandler MSMQ.ReceiveCompleted, AddressOf MSMQIncomingMessage

        RichTextBox1.Text = RichTextBox1.Text = vbCrLf + "Starting" + vbCrLf

        MSMQ.BeginReceive()

    End Sub

    Private Sub MSMQIncomingMessage(ByVal [source] As [Object], ByVal asyncResult As ReceiveCompletedEventArgs)

        Dim mq As MessageQueue = CType([source], MessageQueue)

        Dim AppSpecId As Integer

        Dim m As Message = mq.EndReceive(asyncResult.AsyncResult)

        'Remember how I said I use AppSpecific to control processing of different types of messages?

        AppSpecId = m.AppSpecific

        Select Case AppSpecId

 

            Case 1

                'Nothing this is a ping in my universe.

                m.Dispose()

                mq.BeginReceive()

            Case 999 ' In my universe, AppSpecId=0 tells me to stop processing messages.  I.e. break the cycle, do not continue to receive messages.

                m.Dispose()

                RichTextBox1.Text = RichTextBox1.Text = vbCrLf + "Stopping" + vbCrLf

                Exit Sub

            Case Else

                ProcessIncomingMessages(m)

                m.Dispose()

                mq.BeginReceive()

        End Select

 

 

    End Sub

    Private Sub ProcessIncomingMessages(ByRef Msg As System.Messaging.Message)

        'I have pulled back all the properties above.  I can now read them and do something with them.

 

        'I happen to know this message is going to be a string.

        Dim targetTypes(0) As Type

        targetTypes(0) = GetType(String)

        Msg.Formatter = New XmlMessageFormatter(targetTypes)

        Dim MyBody As String = CStr(Msg.Body)

        Dim sendingId As New SecurityIdentifier(Msg.SenderId, 0)

        Dim MessageFrom As String = sendingId.Translate(GetType(System.Security.Principal.NTAccount)).ToString

        Dim MessageLabel As String = Msg.Label

        MessageFrom = MessageFrom.Substring(MessageFrom.IndexOf("\") + 1)

 

 

        Dim RespondToPath As String

        If Not Msg.ResponseQueue Is Nothing Then

            RespondToPath = Msg.ResponseQueue.MachineName + "\" + Msg.ResponseQueue.QueueName

        End If

 

        RichTextBox1.Text = RichTextBox1.Text + vbCrLf + "New message from: " + MessageFrom + vbCrLf

        RichTextBox1.Text = RichTextBox1.Text + MyBody + vbCrLf

 

 

 

    End Sub

 

    Private Sub StopButton_Click(sender As Object, e As EventArgs) Handles StopButton.Click

        'OK, this is a biggy.  This is what freaks out asynchronous users all the time.  How do you stop the asynchronous listening, short of exiting the program?

        'You have to send your listener a stop message that has a very high priority (moves to the front of the queue) and then you process that message and don't restart beginrecive.

        Dim StopMessage As New Message

        StopMessage.Priority = 7

        StopMessage.AppSpecific = 999

        StopMessage.Label = "I don't have to put anything here, all I care about is AppSpecific=999"

        MSMQ.Send(StopMessage)

    End Sub

End Class

 

Now the sending program. 

The form looks about the same, we just add an extra text box that has the name of the user we want to send to and I have a Send Button instead of Start and Stop.  I include a search for a “\”  in case you want to send it to a different computer.

The only thing you need other than the Imports is the button click code:

  Private Sub SendButton_Click(sender As Object, e As EventArgs) Handles SendButton.Click

        If SendTo.Text = "" Then

            Exit Sub

        End If

        Dim Destination As String = SendTo.Text

        'I am going to assume that if there is a slash in the SendTo text box, that it's in the form ComputerName\UserName

        If Not Destination.Contains("\") Then

            Destination = My.Computer.Name + "\" + Destination

        End If

        If Not System.Messaging.MessageQueue.Exists(Destination) Then

            RichTextBox1.Text = RichTextBox1.Text + vbCrLf + "No such destination."

        Else

            Dim MyMsg As New Message(RichTextBox2.Text) 'Quick hint, the default is a string and you don't have to mess with a Message formatter.

            Dim DestQueue As New MessageQueue(Destination)

            MyMsg.AppSpecific = 5 'Case Else on receive.

            DestQueue.Send(MyMsg)

            MyMsg.Dispose()

            DestQueue.Close()

            DestQueue.Dispose()

 

        End If

 

    End Sub

 

OK, so we wrote a chat program, so what?

Well, this is just a very simple example that should be easily translatable to things in the rest of the world.  Where MSMQ comes in handy is, yes, in server applications and transaction processing, but also in heavy duty multi-threaded WinForms applications as an alternative to Thread Synchronization.  Or, when you have a suite of desktop applications that need to talk to each other.

 

I will follow up in the future with more posts on MSMQ, but hopefully this will get you interested in exploring.

8 comment(s) so far...


Gravatar

Re: MSMQ: The Indispensable Windows Subsystem Most Developers Have Never Heard Of

This was a great blog post. My boss asked me to look into this as an alternative to Log4Net and MS's Logging Application Block. In terms of speed MSMQ blew the other two a way by 5 to 1. I was rather surprised. I had never heard of it before this. I look forward to your next post.

By James on   3/19/2013 1:49 PM
Gravatar

Re: MSMQ: The Indispensable Windows Subsystem Most Developers Have Never Heard Of

Regarding your code, the following only works with AD active directory:

MSMQ.MessageReadPropertyFilter.SourceMachine

By Marc on   5/23/2013 5:29 AM
Gravatar

Re: MSMQ: The Indispensable Windows Subsystem Most Developers Have Never Heard Of

Yes,

You are correct. There are a number of various properties that only work under A/D, or within the same domain/forest. I.e. things get more interesting when you have to send to foreign domains. Particularly of http/https proxies.

For example, a foreign domain is not going to be able to recognize user or machine principles from the sending domain, most likely. Authentication issues can also get more complex.

For a simple example that is a property I probably should not have included, it's not critical for the example, but it does bring up a good point, which you bring up.

Thanks

Jerry

By Jerry on   5/23/2013 7:06 PM
Gravatar

Re: MSMQ: The Indispensable Windows Subsystem Most Developers Have Never Heard Of

I am interested in some benchmarks that MSMQ can provide. In terms of data processing if I see one immediate downside to MSMQ I would like you to address.

Message Sizes; What if I need to return a large amount of data? MSMQ has 4MB size limit.

Then a more serious concern in terms of heavy load processing functions requested by a webservice API. Do you know of some real world performance results that MSMQ can provide?

1. How fast is MSMQ really? Is it the fastest way to transfer function requests back and forth between a web and application server?

2. Threading queue operations; efficacy?

3.If your queue fills up I believe above the set limit of 4GB messages will begin writing out to the disk and eventually the queue will crash, another limitation I have difficulty accepting that there is not a better option out there.

By Dave on   2/16/2014 1:22 PM
Gravatar

Re: MSMQ: The Indispensable Windows Subsystem Most Developers Have Never Heard Of

Hi,

This is a quick answer off the top of my head.

Benchmarks (and in fact most of the limits you discuss) are a function of the design of the overall system and the hardware you give it. So it's not a simple answer. But as a very simple benchmark, in 1999-2000 using a dual processor Pentium III (2 cores total) and mostly an otherwise normal configuration for a high end workstation, running NT 4.0 a trading system that sent trades via a benchmark sender through MSMQ and into a receiver application that decoded the trade and stuck it in the SQL Server, I could do up to 1000/trades per second with a single listener, dependent on where the SQL Server was. Worst case scenarios were 400 to 600 trades per second, with less optimal configurations.

I.e. SQL Server was the bottleneck. Of course many caveats apply, your mileage could vary.

It really comes down to how you design the system, and what language you use. My sender/listeners were written in Delphi, which is a native application, unlike say .Net, which will slow you down a lot. So in my simple case, I maximized everything for speed. If you need the speed you need natively compiled code such as Delphi, C or C++.

Now, if you want to get complicated, use multiple listeners running on other machines. Once you start distributing things (which is ultimately the goal of all queuing systems) you can really start to crank up the numbers.

You can also then create multiple queues and use a directory service to route messages to the least busy queue that is attached to the system. E.g. run multiple Queue Servers that feed the same application. Have multiple readers per queue, both on each "server" and on multiple servers. Scaling is only limited by your budget.

Yes the 4Mb limit appears to be a problem, but it's actually a feature to improve performance and reliability. If the message size gets too large, you get blocking, i.e. stuff waiting for this huge transfer to finish. Avoiding blocking is one huge reason to use a queuing system in the first place.

However, this is not a problem, you can easily break a large message data packet up into multiple packets and then reassemble at the other side. This is, in fact what TCPIP is doing anyway. You can do any number of things to encode sequence number, total number and a checksum into the message headers so you know if the thing is complete, use subqueues to request resends etc. Or wrap them in a transaction etc.

Giant sizes are a huge problem for all methods. All large http uploads and ftp uploads break things up; or if the logic isn't there to break things up, they limit the maximum upload size in a post. Etc. So large scale data transfer is always an issue, the bigger the transfer, the higher the likelihood of interruptions, blocking and related issues so breaking up is a good idea. Ask anyone that's tried to use a non-resuming ftp client and server that's tried to transfer a 120 Gig file across a 50 Mbps or lower connection.

As far as queue size, that's really only limited by your hardware. Individual Queue Size is limited by the overall System Queue size; up the latter and you can up the individual queues. The only limit is ram/disk size. For persistent queues it's disk and for transient queues: RAM. Use a 64 bit OS, use a dedicated, large, high speed disk for the queue location. I'd say a RAID 10 SSD if you've got the money, or a good SAN.

Threading Queue Operations? I'd need some more info on what you are asking. MSMQ is certainly thread safe. I highly recommend having multiple listeners to each queue if you are going to need to have high volumes or lots of processing. In almost every case, the listener, which processes the data is going to be slower than the queuing system; therefore have multiple threads monitoring the queue.

As far as your queue filling up, that's an issue of how big your memory and hard disk are. A transient queue does what you say, a persistent queue is always writing to disk. If you run out of disk space, you've got an issue, as does anything.

So again, move the queue location to a large fast, dedicated drive; if you are doing transient and have a need for speed pile up the gigabytes of RAM. It's only limited by your budget.

Now, to thread back to your question about is it the fastest way? Prossibly not. I'm sure any number of architects could come up with a custom solution that would do better. But then they'd have to built it, debug it, maintain it, publish an API. And some people do. There are custom systems used in very large data projects like the Large Hadron Collider at CERN. But they have a very big budget and a lot of really good data scientists...

The big point, I think, is bang for the buck. Given that it's very fast, and comes free with Windows it's a very good solution for a lot of things. The thing that really gives it power is that it is asynchronous, so threads don't have to sit around waiting for each other. You don't have your major processing threads tied up on empty cycles waiting for something slower to respond. Each individual component can work at it's own speed, and not wait on other slower things.

For the slower processes, just make more of them on machines with more resources and scale each step resource wise commensurate with the intensity of that steps needs, and you can get a system where each step/stage/etc is operating at peak speed.

And of course, the caveat, not all processes benefit by this sort of solution. However, web applications, being natively stateless (meaning developers have to work to "fake" a stateful application) queuing is generally very well suited to web applications.

By Jerry on   2/16/2014 4:45 PM
Gravatar

Re: MSMQ: The Indispensable Windows Subsystem Most Developers Have Never Heard Of

Amazing post

By max on   3/23/2016 9:42 AM
Gravatar

Re: MSMQ: The Indispensable Windows Subsystem Most Developers Have Never Heard Of

I really never ever have read such an informative post which has totally changed my vision of using windows as I used to go online to find solutions and luckily few times I succeeded but today I have got online writing service through which I can have any query solved at any time anywhere with out anyone's help.

By joeright on   3/23/2016 2:32 PM
Gravatar

Re: MSMQ: The Indispensable Windows Subsystem Most Developers Have Never Heard Of

Thank you Guru, for always been there to solve our problems and giving us the feedback the problems of PhD writing service you have solved for the benefit of the people. I always read the post of you to get something unique out of it.

By Friesess on   3/27/2016 3:31 AM

Your name:
Gravatar Preview
Your email:
(Optional) Email used only to show Gravatar.
Your website:
Title:
Comment:
Security Code
CAPTCHA image
Enter the code shown above in the box below
Add Comment   Cancel 
You must be logged in and have permission to create or edit a blog.