MSMQ: The Indispensable Windows Subsystem Most Developers
Have Never Heard Of
MSMQ, Microsoft Message Queuing has been around since the
days of NT 4.0 and Windows 95, way back to 1997. Initially it was an add on
product to windows, but starting with Windows 2000 it was included in the
server versions of Windows and with Windows XP it was included in the desktop
operating system. It’s a Windows Feature that you can install/enable via the
Windows Components part of Add/Remove Programs (or Add Features in W2K8). Yet
surprisingly few application developers know about it or how to use it to
improve application performance. Tonight we start with an introduction, later
posts will talk about improving desktop application performance (everyone who
blogs about MSMQ talks about server uses of MSMQ)
It’s one of the great secrets of high end data processing
programming. Before MSMQ people would pay a lot of money for MQ Series from
IBM (now called Websphere MQ) or similar programs from other third parties.
(People still do, by the way). However, thanks to Microsoft’s generosity, all
Windows machines get it for free.
Traditionally it’s been used to facilitate communication
between applications and servers running at different speeds and priorities
ensuring no one machine was held up by processing on another. It’s still used
for that, of course, but there are a lot of other things you can do with it, in
particular on the Desktop and on the Web.
So what is it?
Simply put, MSMQ is like an e-mail/FedEx for programs, which
makes MQ Series like an e-mail/UPS for programs. It’s (optionally) guaranteed
delivery of messages between different applications running either on the same,
or different machines.
Unlike real email, MSMQ (and Websphere MQ—implied from now
on) is extremely fast, extremely reliable and extremely secure. The API allows
routines to send and receive messages back and forth via message queues.
Queues are stacks of messages that operate, by default, on a FIFO (First In,
First Out) system. Each routine that wants to talk to other routines has one
or more queues associated with it, and there is often, but not always, a
directory service that allows it to locate other routines (although this is
generally dictated by the programming logic).
So, what are these messages?
Messages are binary blobs of data with a structured layout
that the user/programmer doesn’t have to know about. In the API they appear as
a “Message” object. The message object has a number of properties and methods,
much like an object representation of an e-mail message. Some of the most
useful/obvious ones include:
·
Label
o
The Name/Subject of the message, user defined string
·
Body
o
The Body/Contents of the message. It’s a binary blob, use the
Body Type to determine the format of the message. Traditionally the body is
text or XML, but it can also be a binary stream.
·
Body Type
o
Tells the receiver how to interpret the body. There are a number
of predefined types and one can create custom types.
·
Correlation Id
o
Generally a user generated field to tie this message to some other
message, for example to specify a conversation.
·
Priority
o
An integer between 0 and 7 indicating the priority of the
message, the default is 3. How priority is interpreted is up to the developers
on the sending and receiving side.
·
Message Class
o
What type of message is it: Normal, Acknowledgement, Negative
Acknowledgement, etc. Most people only care about normal messages, but if you
are tracing the delivery of your messages, you’ll want to deal with the others
(think Delivery Receipts)
·
App-Specific
o
An application specific unsigned integer.
o
I use it to define application message types to aid in quickly
processing messages.
·
Extension
o
One of my favorites, a user defined binary field, I think of it a
binary TAG field.
·
Destination Queue
o
Where the message was sent to, or about to be sent to. The “TO:”
field, where the address is the pathname of the queue.
·
Response Queue
o
Optionally where any replies should be sent to. The “REPLY-TO:”
field.
·
Sender ID
o
The UID of the person or application sending the message. Used
for authentication purposes (there are also Authentication fields, Encryption
fields etc.)
§
If I have user desktop applications sending MSMQ messages, I’d
have the UID of the user in this field to determine if they are authorized to
request my receiving application do whatever it is the user has asked.
§
This is different than permission on the queue itself, which can
also be set. I may want to allow a wide range of users to submit messages to
this queue, I just may wish to limit which types of “requests” those messages
might contain, so I check the Users Windows Id.
·
Delivery
o
This one gets overlooked, to many people’s dismay.
o
Set this property to MQMSG_DELIVERY_RECOVERABLE to ensure that
when MSMQ gets restarted, the message stays in the queue.
o
The other option MQMSG_DELIVERY_EXPRESS creates non-persistent
messages that go away when MSMQ restarts. This is the default setting. There
are good reasons for this, but I won’t have time to go into them here.
What are Queues?
This was a lot
harder to explain to people 15 years ago, but…
The best way to
understand a queue is by analogy to something modern workers are all familiar
with. An e-mail Inbox. Imagine if you will, that your job, or part of your
job is responding to messages that come into your Inbox (or maybe a set of
Inboxes/Public Folders)? You have to at least look at every message and decide
what action to take, if any, and what response you must provide, if any. And
as long as you are working you have to do this.
Now, imagine you
are an application, a Message Queue is an inbox that you are responsible for
monitoring. You might have a main Inbox and you might also have a bunch of
other Public Folders that receive email. All of those are Message Queues and
the application has to process any messages that come into the queues
(folders/mailboxes) that are assigned to it.
That’s it. That’s
all there is to it. Basically. Yes you can get complicated with priorities,
dead letter queues, journal queues, NDR’s etc. But all of those have e-mail
equivalents.
Queues are inboxes
of messages, the API provides programs with a way to either Peek or Receive
into the queue. Receive is sort of the default option, it’s basically “give me
the next thing in the queue, the oldest thing.” When you receive, you pull the
message out of the queue, and the message behind it moves to the top. If you
decide you don’t want to deal with it, you put it back into the queue, where it
goes to the end of the line (unless you do something about it).
Peeking on the other
hand is just taking a “peek” at the message without removing it. You can
basically peak your way through all the messages in the queue to get an idea of
what’s in there. There are various routines that build on this to give you
lists sorted by priority, response queue, correlation-id etc.
What properties do queues have? Actually not that many,
here are the major ones:
·
Label
o
A friendly name for the queue
·
Pathname
o
There are different formats for this but basically it’s going to
be \
·
ADS_PATH
o
If you store the queue information in Active Directory Services,
it’s the ADS path.
o
To be clear, it’s an LDAP path. LDAP://MyLDAPServer/CN=MyQueue,CN=msmq,CN=MyComputer,CN=Computers,DC=MyDomain,DC=MyCompany,DC=COM
·
Multicast Address
o
If you want to use multicast to address the queue, this is its
address.
·
PathName_DNS
o
DNS pathname for the queue
·
Base Priority
o
The default priority level of the queue.
·
Journal
o
Specifies whether the queue is journaled.
o
If a queue is journaled, it has a journal queue, which is a queue
that all “received” messages go to. When you “Receive” a message, it vanishes
from the queue. If the queue is journaled, a copy of that removed message goes
into the journal.
·
Type
o
A GUID that identifies what service is associated with the
queue. With a few exceptions, this is user generated, generate a new GUID that’s
unique to your application for this purpose.
o
It’s used so applications can find queues to monitor via the
directory service.
·
Quota
o
Maximum number of messages that can be in a queue, if you want to
put a limit.
·
Transaction
o
Is the Queue Transactional.
o
Here’s the cool part: Messages can be delivered transactionally
to ensure deliver/respond to failure. Use DTS between servers.
Those are the main ones, there are a few others for
encryption and security. However, on top of this there are some innate
properties of queues that are determined at the system level. In particular:
Queues can be either public or private. In general private queues are used
within a single application and single machine and public queues are used for
applications to talk with each other on the same or different machines. This
property defines some of the search criteria for queues when using the
directory service.
How do you Access this stuff?
In .Net use the System.Messaging library, you can also use
many other variants and in particular the COM objects.
Queues can also have Triggers on them. Triggers are actions
defined at the system level that fire when messages arrive in a queue to which
the trigger is attached. I don’t particularly like these things. Again, I’m
old school.
My preference is to either have Windows Services that
monitor queues, or a regular Windows Application that listens to queues.
At its most basic level, sending a message is a simple as
creating a Message object, filling out its properties and then using its send
method to send it to a queue whose name you happen to know. If you don’t know
the name of the queue you can search for it via the API in a number of
different ways. If you know the name of the server the queue is on, you can
request a list of all public queues and look through them. Otherwise you have
to rely on a directory service such as DNS or Active Directory Services to find
your queue. I myself generally build my own custom search logic via name
patterns and lookup records. This is simply because I’m old fashioned.
Again, at its most basic, receiving a message is simply a
matter of knowing the pathname of the queue your application wants to monitor,
or looking it up, and then Opening that queue and either Peek into the queue,
do a Synchronous Receive (you will lock until a message arrives or there is a
time out) or do an asynchronous BeginReceive with an attached event handler.
In this last case when a message is received the event is
raised and your handler gets the message as an argument and can then process
it. The trick is to remember to call BeginReceive again when you are done
processing.
Of course, the other, unstated trick is to know if
someone/something else is Receiving from the same queue. No different than if
you have two or more people answering the same email.
Some examples?
First we have a simple form that has a Start Button, a Stop
Button and a RichTextBox.
Its purpose is to receive text messages from another user on
the (presumably) the same machine [Change the computer names on each side to
put them on different machines]
Imports System.Messaging
Imports System.Security.Principal
Public Class Form1
Dim MSMQ
As System.Messaging.MessageQueue
Private Sub
StartButton_Click(sender As Object, e As EventArgs) Handles
StartButton.Click
Dim
MyName = My.User.Name
Dim
MyQueueName As String = My.Computer.Name
+ "\" + IIf(MyName.Contains("\"),
MyName.Substring(MyName.IndexOf("\") +
1), MyName) 'Gets rid of the domain part of the username.
If Not
System.Messaging.MessageQueue.Exists(MyQueueName) Then
System.Messaging.MessageQueue.Create(MyQueueName)
End If
MSMQ = New
System.Messaging.MessageQueue(MyQueueName) 'Creates
a new instance of this object and attaches to the queue of this name. Does NOT
create a new queue.
'The next part is optional,
but I want to specify what fields get pulled back automatically when looking at
the queue
MSMQ.MessageReadPropertyFilter.SenderId =
True
MSMQ.MessageReadPropertyFilter.AppSpecific = True
MSMQ.MessageReadPropertyFilter.Label = True
MSMQ.MessageReadPropertyFilter.ArrivedTime = True
MSMQ.MessageReadPropertyFilter.SourceMachine
= True
MSMQ.MessageReadPropertyFilter.Body = True
MSMQ.MessageReadPropertyFilter.Priority =
True
MSMQ.MessageReadPropertyFilter.CorrelationId = True
MSMQ.MessageReadPropertyFilter.Extension
= True
MSMQ.SynchronizingObject = Me
AddHandler
MSMQ.ReceiveCompleted, AddressOf MSMQIncomingMessage
RichTextBox1.Text = RichTextBox1.Text =
vbCrLf + "Starting" + vbCrLf
MSMQ.BeginReceive()
End Sub
Private Sub
MSMQIncomingMessage(ByVal [source] As [Object], ByVal
asyncResult As ReceiveCompletedEventArgs)
Dim mq As MessageQueue = CType([source],
MessageQueue)
Dim
AppSpecId As Integer
Dim m As Message =
mq.EndReceive(asyncResult.AsyncResult)
'Remember how I said I use
AppSpecific to control processing of different types of messages?
AppSpecId = m.AppSpecific
Select Case
AppSpecId
Case 1
'Nothing this is a
ping in my universe.
m.Dispose()
mq.BeginReceive()
Case 999 ' In
my universe, AppSpecId=0 tells me to stop processing messages. I.e. break the
cycle, do not continue to receive messages.
m.Dispose()
RichTextBox1.Text =
RichTextBox1.Text = vbCrLf + "Stopping" +
vbCrLf
Exit Sub
Case Else
ProcessIncomingMessages(m)
m.Dispose()
mq.BeginReceive()
End Select
End Sub
Private Sub
ProcessIncomingMessages(ByRef Msg As
System.Messaging.Message)
'I have pulled back all the
properties above. I can now read them and do something with them.
'I happen to know this
message is going to be a string.
Dim
targetTypes(0) As Type
targetTypes(0) = GetType(String)
Msg.Formatter = New XmlMessageFormatter(targetTypes)
Dim
MyBody As String = CStr(Msg.Body)
Dim
sendingId As New SecurityIdentifier(Msg.SenderId,
0)
Dim
MessageFrom As String =
sendingId.Translate(GetType(System.Security.Principal.NTAccount)).ToString
Dim
MessageLabel As String =
Msg.Label
MessageFrom =
MessageFrom.Substring(MessageFrom.IndexOf("\") +
1)
Dim
RespondToPath As String
If Not
Msg.ResponseQueue Is Nothing Then
RespondToPath =
Msg.ResponseQueue.MachineName + "\" +
Msg.ResponseQueue.QueueName
End If
RichTextBox1.Text = RichTextBox1.Text +
vbCrLf + "New message from: " + MessageFrom + vbCrLf
RichTextBox1.Text = RichTextBox1.Text +
MyBody + vbCrLf
End Sub
Private Sub
StopButton_Click(sender As Object, e As EventArgs) Handles
StopButton.Click
'OK, this is a biggy. This
is what freaks out asynchronous users all the time. How do you stop the
asynchronous listening, short of exiting the program?
'You have to send your
listener a stop message that has a very high priority (moves to the front of
the queue) and then you process that message and don't restart beginrecive.
Dim
StopMessage As New Message
StopMessage.Priority = 7
StopMessage.AppSpecific = 999
StopMessage.Label = "I
don't have to put anything here, all I care about is AppSpecific=999"
MSMQ.Send(StopMessage)
End Sub
End Class
Now the sending program.
The form looks about the same, we just add an extra text box
that has the name of the user we want to send to and I have a Send Button
instead of Start and Stop. I include a search for a “\” in case you want to
send it to a different computer.
The only thing you need other than the Imports is the button
click code:
Private Sub
SendButton_Click(sender As Object, e As EventArgs) Handles
SendButton.Click
If
SendTo.Text = "" Then
Exit Sub
End If
Dim
Destination As String =
SendTo.Text
'I am going to assume that
if there is a slash in the SendTo text box, that it's in the form
ComputerName\UserName
If Not
Destination.Contains("\") Then
Destination = My.Computer.Name
+ "\" + Destination
End If
If Not System.Messaging.MessageQueue.Exists(Destination)
Then
RichTextBox1.Text = RichTextBox1.Text
+ vbCrLf + "No such destination."
Else
Dim
MyMsg As New Message(RichTextBox2.Text)
'Quick hint, the default is a string and you don't have to
mess with a Message formatter.
Dim
DestQueue As New MessageQueue(Destination)
MyMsg.AppSpecific = 5 'Case
Else on receive.
DestQueue.Send(MyMsg)
MyMsg.Dispose()
DestQueue.Close()
DestQueue.Dispose()
End If
End Sub
OK, so we wrote a chat program, so what?
Well, this is just a very simple example that should be
easily translatable to things in the rest of the world. Where MSMQ comes in
handy is, yes, in server applications and transaction processing, but also in
heavy duty multi-threaded WinForms applications as an alternative to Thread
Synchronization. Or, when you have a suite of desktop applications that need
to talk to each other.
I will follow up in the future with more posts on MSMQ, but
hopefully this will get you interested in exploring.