IBM WebSphere Message Broker introduced Record and Replay as part of its version 8 release. I thought I would just write a brief quickstart guide to getting running with record and replay, and . . . → Read More: Message Broker v8: Record, Replay and REST]]> IBM WebSphere Message Broker v8 was released in December last year – There is an announcement and comparison of versions on the IBM website.

IBM WebSphere Message Broker introduced Record and Replay as part of its version 8 release. I thought I would just write a brief quickstart guide to getting running with record and replay, and in particular with working with the new REST interface which comes with it. There is actually a Record and Replay specific PDF available too, which contains all the details you might want for using Record and Replay (far more detail than I cover here). The main reason I am writing this, is it can be a little daunting trying to get Record and Replay going, as it does have a fair bit of configuration, and involves a few areas of a quite large product.

Setting up Recording

From your WMB install dir, run the db2 schema file to create the sample database for Record and Replay. Edit the file if you want to use a database name other than MB8RECORD, a specific schema or increase the storage size per message above 5mb.

db2 -tvf ddl/db2/DataCaptureSchema.sql

Set the password for your database

mqsisetdbparms MB8BROKER -n MBRECORD -u username -p password

Set the DataCaptureStore configurable service to set where to record to and which EG to do the recording (You can also do any of these configurable service steps via the Message Broker Explorer gui)

mqsicreateconfigurableservice MB8BROKER -c DataCaptureStore -o MyCaptureStore \
   -n "dataSourceName,egForRecord" -v "MBRECORD,default"

Define where to record messages from. Use a wildcard to set all configurable services. Note that double quotes (“”) are required on unix, windows will want single quotes (”)

mqsicreateconfigurableservice MB8BROKER -c DataCaptureSource -o MyCaptureSource \
   -n "dataCaptureStore,topic" -v "MyCaptureStore,$SYS/Broker/MB8BROKER/Monitoring/#"

Alternatively you can list the flows explicitly with topics like “$SYS/Broker/MB8BROKER/Monitoring/ExecutionGroupName/FlowName” While we are configuring everything, set a replay destination too. Set EG default to replay to queue “REPLAY” on queue manager MB8QMGR

mqsicreateconfigurableservice MB8BROKER -c DataDestination -o MyDestination \
   -n "egForReplay,endpoint,endpointType" -v "default,wmq:/msg/queue/REPLAY@MB8QMGR,WMQDestination"

Setting up Monitoring

Now your broker is configured for recording, you will need to setup something to be recorded. Event Monitoring is used to emit data to record, so either follow the monitoring sample, read the docs on Event Monitoring or edit the sample monitoring profile in sample/RecordReplay/basicMonitoringProfile.xml to replace NODENAME with the name of your flow’s input node. Of course you can also configure event monitoring from within the flow editor in the toolkit – simply select a node and you will find monitoring in the properties view. If you have the flow source already, it is much easier to set the monitoring in the toolkit than it is to do it via the command line. You can also then use the mqsireportflowmonitoring command afterwards to give you the xml monitoring profile being used (which is basically what you are setting in the toolkit when setting the monitoring options on a flow)

There are two specific properties which are worth mentioning when it comes to monitoring for Record and Replay.
If you tick the include bitstream option in the toolkit then the body of your message will be recorded. Choose a sensible encoding, as if you select CDATA and your body will not be a valid XML element, you will break the monitoring and no event will be emitted. If in doubt, use the hexbinary or base64 encoding. It is worth saying that if you record MQ headers, and replay that message, the headers will be attempted to be reused. Other transport headers will just end up in the body of your message.
The other option I wanted to mention was the UserData area of Event Monitoring. If you include $ExceptionList in your monitoring settings, then if an error occurs and an event is emitted, the exception list will be included in the monitoring message. This will be recorded, and will be available from the Record and Replay web interface and REST Api.

If you used a monitoring profile you will need to configure the profile with the following commands:

mqsicreateconfigurableservice MB8BROKER -c MonitoringProfiles -o MyProfile

mqsichangeconfigurableservice MB8BROKER -c MonitoringProfiles -o MyProfile \
   -n profileProperties -p "sample/RecordReplay/basicMonitoringProfile.xml"

mqsichangeflowmonitoring MB8BROKER -e default -f MyFlow -m  MyProfile

Monitoring normally defaults to off, so you will probably need to turn it on with:

mqsichangeflowmonitoring MB8BROKER -e default -f MyFlow -c active

You can check that monitoring is correctly active with the command

mqsireportflowmonitoring MB8BROKER -n -e default -f MyFlow

You should see output like

BIP8911I: Monitoring settings for flow 'MyFlow' in execution group 'default' -  State?: active, ProfileName: 'MyProfile'.
BIP8912I: Event: 'IN.transaction.End',  Event name: 'IN.transaction.End',  Configured?: yes,  State?: enabled.
BIP8912I: Event: 'IN.transaction.Rollback',  Event name: 'IN.transaction.Rollback',  Configured?: yes,  State?: enabled.

You can now put messages through your flow and they will be recorded.

Enabling the REST interface and Web UI

To enable the REST interface and Web UI you need to enable the HTTP server and set the port for it to run on. You will also need to set which ExecutionGroup will process requests to view recorded messages. (You can also do this via the MBX properties, see the properties for your broker, and for your DataCaptureStore ConfigurableService)

mqsichangeproperties MB8BROKER -b webadmin -o server -n enabled -v true
mqsichangeproperties MB8BROKER -b webadmin -o HTTPConnector -n port -v 8080
mqsichangeproperties MB8BROKER -o DataCaptureStore -c MyCaptureStore -n egForView -v default

You can also configure HTTPS if you prefer, details are in the product documentation.

Viewing recorded data

You can of course view all the data via a web browser by pointing to the url http://localhost:8080 but you can also do it all via the command line. The following will get back the most recent 30 recorded pieces of data from your captureStore MyCaptureStore.

$curl "http://localhost:8080/datacapturestore/MyCaptureStore/"

This returns quite a large XML list of messages, so let’s try getting just one message.

$curl "http://localhost:8080/datacapturestore/DefaultCaptureStore/?numberOfEntriesPerPage=1"
<?xml version="1.0" encoding="UTF-8"?>
<DataCaptureStore TotalResults="92" NumberOfEntriesPerPage="1" PageNumber="1">
<DataCaptureEntry wmb_msgkey="414d51204d4238514d475220202020203d1bc14e4d290020:414d51204d4238514d475220202020203d1bc14e1b3a0020"
  has_bitstream="Y" has_exception="N" has_userdata="N" event_type="IN.transaction.End"
  event_name="IN.transaction.End" event_srcaddr="IN.transaction.End" broker_name="MB8BROKER"
  broker_uuid="d405fc08-0ec6-11e1-9d9b-7f0000010000" exgrp_name="default" exgrp_uuid="7fd654a2-3301-0000-0080-d4d9103989f9"
  msgflow_name="MyFlow" msgflow_uuid="0a4555a2-3301-0000-0080-e2ed33c9dfcd" appl_name="" appl_uuid="" library_name=""
  library_uuid="" node_name="IN" node_type="ComIbmMQInputNode" detail="IN" terminal_name="" key_fld_1_nm=""
  key_fld_2_nm="" key_fld_3_nm="" key_fld_4_nm="" key_fld_5_nm="" event_timestamp="2011-11-16 11:25:00.647"
  local_transaction_id="9f228bde-1045-11e1-9fac-7f0000010000-1" parent_transaction_id="" global_transaction_id=""/>
</DataCaptureStore>

You can of course filter on any of the data fields above as follows.

$curl "http://localhost:8080/datacapturestore/DefaultCaptureStore/?numberOfEntriesPerPage=1&has_bitstream=Y"

Filtering is possible on all the above columns, with the addition that you can also set a timestamp range with The REST Api has documentation at docs/REST/index.html which covers all these options.

You can also query which DataCaptureStores are setup for viewing by loading the base url “http://localhost:8080/datacapturestore/”

You may then want to download the body of a recorded message. (My example has a shortened url as the message key rather long. Use the wmb_msgkey from your above list requests) Depending on how you set the data to be recorded in your monitoring profile, you will get back a different encoding data

$curl "http://localhost:8080/datacapturestore/DefaultCaptureStore/12345"
<Data encoding="base64Binary">SGVsbG8gV29ybGQK</Data>

You don’t have to get the data back as encoded payloads wrapped in XML. If you set download=true then you will just get the decoded bitstream

$curl "http://localhost:8080/datacapturestore/DefaultCaptureStore/12345?download=true"
HelloWorld

You can also download an ExceptionList if your message has one recorded; set payload=exceptionlist to get the exceptionlist instead of the bitstream. This can also be combined with download=true

$curl "http://localhost:8080/datacapturestore/DefaultCaptureStore/12345?payload=exceptionlist&download=true"

Replaying recorded data

As we already defined a DataDestination called MyDestination above, we have somewhere we can replay to. Again, you can do this via the web browser quite easily, but doing it with curl isn’t hard either.

$curl "http://localhost:8080/datacapturestore/DefaultCaptureStore/12345" -d "replaydestination=MyDestination"

Viewing more about your broker

There are lots of other details about your broker and what is deployed on it available via “http://localhost:8080/admin”. See the REST Api documentation for full details

$curl "http://localhost:7101/admin"
<?xml version="1.0"?>
<brokerResponse brokerName="MB8BROKER" completionCode="ok">
<executiongroup name="default" runstate="running" path="/admin/executiongroup/default"></executiongroup>
<executiongroup name="test" runstate="running" path="/admin/executiongroup/test"></executiongroup>
<log name="Log" runstate="" path="/admin/log/Log"></log>
<adminqueue name="Administration Queue" runstate="" path="/admin/adminqueue/Administration Queue"></adminqueue>
</brokerResponse>

$curl "http://localhost:7101/admin/executiongroup/test/"
<?xml version="1.0"?>
<brokerResponse brokerName="MB8BROKER" completionCode="ok">
<application name="TestXpath" runstate="running" path="/admin/executiongroup/test/application/TestXpath"></application>
</brokerResponse>

$curl "http://localhost:7101/admin/executiongroup/test/properties"
<?xml version="1.0"?>
<brokerResponse brokerName="MB8BROKER" completionCode="ok">
<Name>test</Name><ShortDescription></ShortDescription><LongDescription></LongDescription>
<Type>ExecutionGroup</Type><UUID>3adf66ca-3301-0000-0080-814338bb17bb</UUID><FlowDebugPort>0</FlowDebugPort>
<TraceNodes>on</TraceNodes><Running>true</Running>
</brokerResponse>

“http://localhost:7101/admin/executiongroup/test/extendedproperties” will return a very large amount of information.

Several /admin paths also support recurse=true, for example to see everything deployed on your broker use ”http://localhost:8080/admin?recurse=true”

This was only a brief introduction to Record and Replay in WMB 8, and there is much more detail in the normal documentation. Hopefully a future update to WMB will bring more power and functionality to this area, as well as some more database support. Obviously I can’t say what will be added, let alone when, but I hope you get some value out of what is there currently.

edit 16/01/12 – corrections as supplied by Sebastian

IBM WebSphere Message Broker now uses UnixODBC to connect to many databases. Though not supported by IBM, . . . → Read More: Message Broker v8 with MySQL]]> Now that you know that you can use Message Broker v8 on Ubuntu for development, you may be wondering if you can use some of the databases that are more common on Linux, such as MySQL. You can!

IBM WebSphere Message Broker now uses UnixODBC to connect to many databases. Though not supported by IBM, you can actually connect to MySQL from Message Broker using UnixODBC.

If you are on Ubuntu, you can install the MySQL UnixODBC dirver and the MySQL server with apt-get:

apt-get install libmyodbc mysql-server

Your environment variable $ODBCINI should point to your odbc.ini, mine points to /var/mqsi/odbc/odbc.ini and the base file was copied from /opt/IBM/mqsi/8.0.0.0/ODBC/unixodbc/odbc.ini

You need to set a database definition for a mysql database, something like the following (assuming “wmb” is the name of your MySQL database created later)

[MYSQL]
Description     = MYSQL DB
Driver          = MySQL
Database        = wmb
Server          = 127.0.0.1
Port            = 3306

You also need to make sure your $ODBCSYSINI points to the folder (not the file) where your odbcinst.ini file has been copied to. Mine is set to /var/mqsi/odbc and the odbcinst.ini file was copied from /opt/mqsi/8.0.0.0/ODBC/unixodbc/odbcinst.ini, you also need to add a definition to explain to UnixODBC how to connect to a mysql database

The following should be enough to get UnixODBC working with MySQL

#Sample from package libmyodbc
[MySQL]
Description     = MySQL driver
Driver          = /usr/lib/odbc/libmyodbc.so
Setup           = /usr/lib/odbc/libodbcmyS.so
CPTimeout       =
CPReuse         =

Now create a mysql database (mysqladmin create wmb). You may need to GRANT some permissions to your user in MySQL, or add yourself to the mysql group and logout and back in again.

You can now check the database connection using the WMB mqsicvp command, which will also describe which database functions are available:

mqsicvp -n MYSQL -u anton -p password

BIP8270I: Connected to Datasource 'MYSQL' as user 'anton'. The datasource platform is 'MySQL', version '5.1.58-1ubuntu1'.
===========================
databaseProviderVersion      = 5.1.58-1ubuntu1
driverVersion                = 05.01.0006
driverOdbcVersion            = 03.51
driverManagerVersion         = 03.52.0002.0002
driverManagerOdbcVersion     = 03.52
databaseProviderName         = MySQL
datasourceServerName         = 127.0.0.1 via TCP/IP
databaseName                 = wmb
odbcDatasourceName           = MYSQL
driverName                   = libmyodbc5.so
supportsStoredProcedures     = Yes
procedureTerm                = stored procedure
accessibleTables             = No
accessibleProcedures         = No
identifierQuote              = `
specialCharacters            = None
describeParameter            = No
schemaTerm                   =
tableTerm                    = table
sqlSubqueries                = 31
activeEnvironments           = 0
maxDriverConnections         = 0
maxCatalogNameLength         = 192
maxColumnNameLength          = 192
maxSchemaNameLength          = 0
maxStatementLength           = 8192
maxTableNameLength           = 192
supportsDecimalType          = Yes
supportsDateType             = Yes
supportsTimeType             = No
supportsTimeStampType        = No
supportsIntervalType         = No
supportsAbsFunction          = Yes
supportsAcosFunction         = Yes
supportsAsinFunction         = Yes
supportsAtanFunction         = Yes
supportsAtan2Function        = Yes
supportsCeilingFunction      = Yes
supportsCosFunction          = Yes
supportsCotFunction          = Yes
supportsDegreesFunction      = Yes
supportsExpFunction          = Yes
supportsFloorFunction        = Yes
supportsLogFunction          = Yes
supportsLog10Function        = Yes
supportsModFunction          = Yes
supportsPiFunction           = Yes
supportsPowerFunction        = Yes
supportsRadiansFunction      = Yes
supportsRandFunction         = Yes
supportsRoundFunction        = Yes
supportsSignFunction         = Yes
supportsSinFunction          = Yes
supportsSqrtFunction         = Yes
supportsTanFunction          = Yes
supportsTruncateFunction     = Yes
supportsConcatFunction       = Yes
supportsInsertFunction       = Yes
supportsLcaseFunction        = Yes
supportsLeftFunction         = Yes
supportsLengthFunction       = Yes
supportsLTrimFunction        = Yes
supportsPositionFunction     = Yes
supportsRepeatFunction       = Yes
supportsReplaceFunction      = Yes
supportsRightFunction        = Yes
supportsRTrimFunction        = Yes
supportsSpaceFunction        = Yes
supportsSubstringFunction    = Yes
supportsUcaseFunction        = Yes
supportsExtractFunction      = Yes
supportsCaseExpression       = Yes
supportsCastFunction         = Yes
supportsCoalesceFunction     = Yes
supportsNullIfFunction       = Yes
supportsConvertFunction      = No
supportsSumFunction          = Yes
supportsMaxFunction          = Yes
supportsMinFunction          = Yes
supportsCountFunction        = Yes
supportsBetweenPredicate     = Yes
supportsExistsPredicate      = Yes
supportsInPredicate          = Yes
supportsLikePredicate        = Yes
supportsNullPredicate        = Yes
supportsNotNullPredicate     = Yes
supportsLikeEscapeClause     = Yes
supportsClobType             = Yes
supportsBlobType             = Yes
charDatatypeName             = char
varCharDatatypeName          = varchar
longVarCharDatatypeName      = long varchar
clobDatatypeName             = N/A
timeStampDatatypeName        = N/A
binaryDatatypeName           = binary
varBinaryDatatypeName        = varbinary
longVarBinaryDatatypeName    = long varbinary
blobDatatypeName             = N/A
intDatatypeName              = integer
doubleDatatypeName           = double
varCharMaxLength             = 0
longVarCharMaxLength         = 0
clobMaxLength                = 0
varBinaryMaxLength           = 0
longVarBinaryMaxLength       = 0
blobMaxLength                = 0
timeStampMaxLength           = 0
identifierCase               = Mixed
escapeCharacter              = \
longVarCharDatatype          = -1
clobDatatype                 = 0
longVarBinaryDatatype        = -4
blobDatatype                 = 0

BIP8273I: The following datatypes and functions are not natively 
supported by datasource 'MYSQL' using this ODBC driver:
Unsupported datatypes: 'TIME, TIMESTAMP, INTERVAL' Unsupported functions: 'CONVERT'

Examine the specific datatypes and functions not supported
natively by this datasource using this ODBC driver.
When using these datatypes and functions within ESQL, the
associated data processing is done within WebSphere Message
Broker rather than being processed by the database provider.  

Note that "functions" within this message can refer to functions or predicates. 

BIP8071I: Successful command completion.

There – done! Now you can use MySQL with WebSphere Message Broker on Ubuntu. Don’t forget to set the database password with mqsisetdbparms before trying to actually connect from your database nodes.

Also, the name of the database according to WMB is “MYSQL” as that is what I defined in the odbc.ini. The name of the database according to mysql is “wmb”. They don’t need to be different, but it might help having unique names if you are trying to track down a database connection issue.

Of course, the mysql database server does not need to be local to the broker – The mysql odbc driver should allow you connect to any accessible tcpip mysql server.

I have been running MQ and Message Broker on Ubuntu and Debian since shortly after I joined IBM in 2005, and it seems there are lots of other people doing this too despite it not being a supported platform before . . . → Read More: WebSphere Message Broker on Ubuntu]]> WebSphere Message Broker v8 now supports Ubuntu for development systems (i.e. not production use) - http://www-01.ibm.com/support/docview.wss?uid=swg27023600#Ubuntu

I have been running MQ and Message Broker on Ubuntu and Debian since shortly after I joined IBM in 2005, and it seems there are lots of other people doing this too despite it not being a supported platform before now.

Lots of people have advice on how to install MQ and WMB, and it is worth mentioning them in case you have problems.

http://www.reidmorrison.com/linux/install-websphere-mq-on-ubuntu

http://www.gefira.pl/blog/2010/07/03/websphere-mq-and-ubuntu-howto/

http://blog.herbert.groot.jebbink.nl/2009/06/wmb-61-ubuntu-904.html

The best advice I can give for installing MQ and WMB on Ubuntu is:

• change the /bin/sh symlink to point to /bin/bash – MQ Doesn’t like installing with dash as the default shell.

• use rpm to install MQ – Alien is a bit of a hack, and does not work well. You will need to use the “–force-debian” flag on rpm to make it install.

• One other thing which might help is to run the mqlicense.sh script with the ‘-console’ flag as it may not find your X applications properly.

Some user’s have noticed that chown on Debian and Ubuntu strips the setuid bit from the binaries (Debian and Ubuntu consider leaving setuid set on an executable when you change it’s owner a security flaw, whereas Redhat and SuSE appear not to) so you may need to fix the permissions (best to check the permissions of the same level of MQ from a RHEL or SLES box and set them the same) though I have not seen this with recent versions of MQ.

Message Broker v8 installs quite happily on Ubuntu. The only issues that I know of are that some of the eclipse based gui applications do not draw everything correctly. This is a known eclipse-GTK bug, and is more common on releases after Lucid Lynx (10.04). A workaround is to set the environment variable  GDK_NATIVE_WINDOWS=1

Update 05/01/12 – I have just noticed that the script ‘mqsicreateworkpath’ which is used to initialise /var/mqsi correctly still uses ksh. Either install ksh on your system or edit the script to say bash in the first line instead of ksh (it should work fine then)

Fix Pack 7.0.0.1 (V7.0 Fix Pack 1) for WebSphere Message Broker v7.0 is now available. http://www-01.ibm.com/support/docview.wss?&uid=swg24027267

You can see what is new in 7.0.0.1 by looking in the Information Center

The new WebSphere Message Broker Accounting and Statistics facility can help you monitor message flow performance and resource usage of a broker or execution group at the message flow, node, or terminal level, . . . → Read More: Monitoring message flow performance with IS02]]> My DeveloperWorks article on Monitoring message flow performance with the WebSphere Message Broker Explorer Accounting and Statistics facility has finally been published!

The new WebSphere Message Broker Accounting and Statistics facility can help you monitor message flow performance and resource usage of a broker or execution group at the message flow, node, or terminal level, and can thus be a tremendous help in solving performance and resource utilization problems.
Hopefully you will find it a good starting guide for working with the new Accounting and Statistics views in IS02

Full details in the official . . . → Read More: IBM SOA 6.1 range announced]]> I had meant all along to blog a little about work, so to reiterate a post from SOA Tips ‘n’ Tricks, the Websphere SOA 6.1 range has been announced by IBM: Websphere Message Broker, WebSphere Enterprise Service Bus, WebSphere Process Server, WebSphere Message Broker, the WebSphere Service Registry and Repository.

Full details in the official release letter

It provides a single point for administering your Message Brokers and can be used side by side with your current Message Broker Toolkit. The Broker Explorer (IS02) has the following features…