Sunday, December 28, 2014

Domino Websocket Fixes & Performance testing for v1.1.3

***websocket project has been broken out into its own project, on openntf***

The latest release of Domino Websockets v1.1.3 focuses on bug fixes and performance.  Most notable fixes include:

  • Fixed memory leak for users with closed websocket connections.  This was causing java.lang.OutOfMemory errors during consecutive test runs, and left users in an inconsistent state in the backing ConcurrentHashMap and user document.

  • Increased routing performance by only serializing the routing attributes of SocketMessage, and only making the serialization call once. This also helped to eliminate java.lang.OutOfMemoryErrors.

  • Cleaned up the SSJS Rhino client implementation to now render the script path in the user profile instead of just a UUID.

  • Fixed problem when transitioning from Anonymous to Authenticated users that already have a websocket session established.

  • Added the following notes.ini parameters:
    • WEBSOCKET_THREAD_COUNT defaults to 2 (This only impacts threads loaded for routing. Websocket workers are still equivalent to the number of cores / CPUs on the host machine.)
    • WEBSOCKET_ARRAY_QUEUE defaults to false (allows admin to change the internal data structure from LinkedBlockingQueue to ArrayBlockingQueue.  No performance advantages were seen during testing.  This was added to see if a bound blocking queue would reduce the memory footprint during JUnit test runs.  Performance results were inconclusive)
    • WEBSOCKET_ARRAY_QUEUE_SIZE defaults to 10k (if WEB_SOCKET_ARRAY_QUEUE is set to true, uses this value to set the upper bound of the array blocking queue).
    • WEBSOCKET_PURGE_INTERVAL defaults to 900 sec (instead of relying on agents to cleanup expired messages and users).

Latest benchmarks for performance are:
  • 2000+ concurrent users during JUnit test runs

  • 1 million messages sent via 'Broadcast Many' agent in approximately one minute (loop back address used so no network latency, and message was only 270 bytes).

    • VariableSizeWebSocket load test pushed messages varying in size from 50kb to 25mb without any OutOfMemory errors.  Transferred about 12gb of data over 15 minutes to 2000 clients.

    Testing Environment:
    • Both test client and server were running off the same machine (localhost used, no network latency).
    • -Xms1024M -Xmx4096M for client execution
    • Hardware:
      • MacBook Pro
      • 16gb RAM
      • SSD (Solid State Hard Drive)
      • Core i7 (quad core processor)
    • Relevant notes.ini settings were
      • HTTPJVMMaxHeapSize=4096M

    To re-create tests see plugin project.

    FYI, I haven't had time to test the last few releases across a Domino server cluster (requires borrowing my boys gaming rig... they aren't too happy as I typically borrow it for a week at a time :). That is on my ToDo list for next year.  If you set this up on a cluster, please let me know your results.... thanks for stopping by!

    download from OpenNTF


    Wednesday, December 10, 2014

    Domino WebSockets & Dashboards... it's like Christmas :)

    ***websocket project has been broken out into its own project, on openntf***
    Hi All,
    Getting close to completing the dashboard application that consumes and renders state information from remote devices / clients via websockets.  As I was building it I couldn't help but notice how festive the colors were.... so I decided to create a quick video of the application accompanied by my favorite Christmas song "Christmas Wrapping" by the Waitresses (links provided to appease the copyright gods).  Normally the app would not overwrite any errors (red), or warnings (yellow), but I disabled that logic to keep the colors going throughout the song :).  The JUnit test randomly selects the target location of the device (room number in this case), and also randomly selects the socket message data (error, warning, OK).  The client side JavaScript receives the message, then queries the DOM for the location and alters the .css class reference to change the color.  In addition to the UI, you should also notice the RhinoClient running in the console updating the documents on the back-end if an error or warning is sent in from the device.  Anyway... I guess I hit the eggnog a little early this year... Merry Christmas!

    link to video

    Wednesday, November 12, 2014

    SSJS and Websockets

    ***websocket project has been broken out into its own project, on openntf***

    A new use case recently emerged during my latest development efforts to build a websocket based application / dashboard.  In addition to the requestURI routing path feature that updates all clients via websocket in the UI, I also needed to execute some server side logic for any messages coming in through that channel.  I thought of a few options:

    • Have the client side web browser that is behaving as the visual dashboard post the data back to the server via Ajax / REST call 
    • Send update via REST call and have the server process and dispatch the message via websocket
    • Load up an embedded client on the same server as the websocket server to receive and process the messages

    I opted for the last approach, and took it a bit further than just loading a single client.  In the next incremental release of the Domino WebSocket plugin, developers will be able to register SSJS libraries to consume / intercept websocket client events via simple API call.  NOTE:  The SSJS is compiled and executed under the Rhino scripting engine, not the XPages version of SSJS.  There are some syntax differences you need to be aware of, namely type declarations (i.e. var str:String = "some string" will not compile must only be var str="some string").  Be sure to omit them when writing SSJS for websockets.  Below are the steps required to register a script library with the websocket plugin:

    1) Create your SSJS script library websocket client (must be of type SSJS when creating lib in designer)

    Below is the output from the SSJS client to any users on /chat.nsf

    2) Register your Rhino compliant SSJS lib a couple of ways... via SSJS API call

    command line

    (note that * covers all events onOpen, onMessage, onClose, and onError)

    A few other important notes about the default objects in scope of your script(s):
    • session (uses server's Id, so make sure the server has appropriate rights to the target nsf housing the script lib)
    • websocketClient  (you can use this object to send messages out in response to incoming events)
    • event (just the name of the event onOpen, onClose, onMessage, onError)
    • socketMessage is only available via onMessage event
    • bundleUtils  (facilitates accessing specific OSGi plugins in the XPage runtime see print method in above code, target class must use no arg constructor)

    New Command line options:

    • register-script (takes four arguments host uri event scriptpath)
    • reload-scripts (takes no arguments and reloads from source nsf and re-compiles all scripts)
    • show-scripts  (takes no arguments, renders all the registered scripts)
    • remove-script (takes one argument that is the path to the lib i.e. /chat.nsf/scriptlib)


    • At some point I might refactor the client out of the websocket plugin into its own to allow SSJS clients access to any remote websocket server (Domino, Node, JavaEE based).

    I'll be pushing this up to github and openntf soon, please check back in a few days.... thanks for taking the time


    latest release available on openntf

    Sunday, September 21, 2014

    Using request URI as a routing path for Domino/XPage websocket messages

    In my current project, I'm going through the process of replacing a bunch of restlet programs with a single webscocket client to enable bi-directional communications with the XWork server and about 300 clients.  As part of my requirements for the app, I need to be able to send targeted websocket messages to users on a specific page to render system / log data on a dashboard to monitor the client state.  I don't want users getting socket messages in an app that is expecting a different type of socket message data.  As a result of this need, I've gone ahead and updated the Domino Websocket plugin to include support for routing messages based on the user's request URI and role.  Below are a few examples of the routing that can be achieved with a simple string in the "to" attribute of the socket message from any of your websocket enabled applications.  See the chat.ntf application for a working copy of the samples below.

    • /chat.nsf*[mgr]    (sends to everyone in the chat application with [mgr] role)
    • /chat.nsf/chat.xsp[mgr],[admin]  (sends to everyone on the specific chat.xsp page with the [mgr] or [admin] role)
    • /chat.nsf  (sends to anyone on the default page of chat.nsf)
    • /chat.nsf* (sends to everyone on any page/xpage in chat.nsf)
    • /chat.nsf/chat.xsp/CN=admin admin/O=marksdev (send direct message to user admin admin on page chat.xsp)

    The request URI is also stored with the user profile created when the websocket session is established, to allow for more complex querying by background agents, SSJS, or Java (see Users by URI view in websocket.ntf).

    download available here webshell-xpages-ext-lib

    Friday, June 13, 2014

    Domino WebSockets now with REST API

    As of release 1.1.0
    -added simple REST API for accessing the websocket server (see websocket-setup.pdf)
    -updated the chat application example to use the new REST APIs (see classic note web form fmchat)
    -updated JUnit tests for REST API, added a java JSON Client (make sure to setup basic auth via website rules)
    -added simplified configuration. notes.ini parameters are only required for special configuration (i.e. clustering), or to override the defaults

    (note: be sure to replace websocket.ntf, and chat.ntf designs on your server, and add websocket to the Domino Access Services in the server or website doc.)

    download available on openntf

    Monday, June 02, 2014

    Latest Updates for Domino WebSocket

    ***websocket project has been broken out into its own project, on openntf*** 
    As of release 1.0.9 for websocket:
    -Modified the plugin to accept command line parameters:
    Available Command Line Operations:
    • tell http osgi websocket stop (Run this prior to stopping http to avoid server crashes. This command will terminate the threads cleanly)
    • tell http osgi websocket start
    • tell http osgi websocket count (gets the current count of websockets on the current server)
    • tell http osgi websocket count-all (gets the count of all the websockets across a cluster)
    • tell http osgi websocket show-all-users (shows all users currently using a websocket across a cluster)
    • tell http osgi websocket show-users (shows users on current server)
    -Added code in all runnables to listen for a stop signal.
    -Added better DoS attack prevention via below notes.ini params:
    • WEBSOCKET_MAX_MSG_SIZE (instead of waiting for entire message, it attempts to capture size incrementally and terminate connections if max)
    • WEBSOCKET_MAX_CONNECTIONS (will terminate connections if max is reached)
    -Refactored the broadcast logic to create less documents. Now it only creates one document per server in a cluster as opposed to one per user.
    -Updated testing code to execute randomly sized messages (see in project)
    -Overall this release should be the most stable.... unless I missed something :)
    download available on openntf

    Monday, May 19, 2014

    Clustering Support in Domino WebSockets now available

    ***websocket project has been broken out into its own project, on openntf***

    Hi All,
    I just released version webshell-xpages-ext-lib 1.0.8 (parent project for websockets).  The latest release focuses on support for websockets in a clustered environment and increases the message size.  Please see my prior blog post for info. on the approach that was taken, and let me know if you exp. any problems bugs via github's issue tracker.  The documentation has been updated with the latest info. on how to setup for clustering.

    As of release 1.0.8 for websocket: 
    -Fixed configuration problems causing http server to hang (websocket server will not load unless configured correctly, better error messages if not configured) 
    -Added @Profiled annotation to all runnables to measure performance via xpages toolbox 
    -Added support for Domino server clusters (all users / apps across clusters can now dispatch messages to each other) 
    -Altered the data structure of the fmSocketMessage document to allow larger messages via RTF append, or direct json attachments 
    -Added some new notes.ini parameters for clustering and broadcasting (see webscket-setup.pdf) 
    -Updated all the existing sample LotusScript agents to use the new data structure 
    -Added two new sample agents “JSON File” & “JSON RichText” to serve as examples on how to use the new document data structure
    below demo is still valid:


    Wednesday, May 14, 2014

    Bug Fixes & Supporting Server Clusters in Domino WebSockets

    ***websocket project has been broken out into its own project, on openntf***

    Hi All,
    Currently working on some bug fixes to make configuring the Domino WebSockets a bit friendlier.  Seems if the notes.ini settings weren't setup properly it could hang up your http server on restart, or the plugin will not work altogether.  Instead the admin / developer should see a nice error message stating the problem with the notes.ini settings.  In addition to the fixes for the config, I'm in the process of adding support for server clusters.  The below list summarizes the approach I am taking in supporting Domino server clusters:

    1)  As part of the IUser/User object I added a host attribute and also persist that value to their fmUser document

    2)  The thread that processes the queues will only process documents that match its current host.

    3)  A new thread is monitoring the user status and updating / adding users to the ConcurrentHashMap that stores the Ids in memory on all servers in the cluster

    4)  Upon receiving a websocket message directly from the client (i.e. in the chat example), the WebSocket server checks to see if the target user is on the same server, if yes, send them direct, if no, drop the message in the websocket.nsf queue to be picked up on the target user's host server via cluster data replication.

    5) Each server in the cluster monitors one other in the cluster via a notes.ini parameter WEBSOCKET_CLUSTERMATE_MONITOR.  If the target clustermate goes down after N seconds configured in WEBSOCKET_CLUSTERMATE_EXPIRATION notes.ini param, the current server will alter the status of the online users to offline.

    6)  Broadcast websocket messages had to be handled differently in a cluster, so I altered the behavior to just create one copy / online user and create a direct message.  In a cluster one server in the clustered environment must be listed as the broadcast server, or no broadcasts will go out.  If the server designated as the broadcast cluster goes down... one of the other servers will have to be manually setup to take over (sucks... will re-think this).

    Below is a sample from my local dev machine configured with a clustermate on my dev centos box.

    #start websocket settings


    #debug/test settings


    #security settings,

    #network encryption settings


    #clustered server settings


    (note:  this has only been run / tested on my clustered dev environment)

    First pass of the above should be up on github and openntf sometime this week.



    Wednesday, April 09, 2014

    Domino WebSocket Updates

    ***websocket project has been broken out into its own project, on openntf***

    Just added some new features to the Domino WebSocket plugin.  Most notably, the ability to queue up SocketMessages to users that have a browser websocket connection established on the server.  Have a look at the LotusScript sample agents that create socket messages.  The below video provides a demo of the sample chat XPage application, and the SocketMessage queues.

    openntf or github



    Monday, March 31, 2014

    Domino Websocket now available

    ***websocket project has been broken out into its own project, on openntf***

    I recently posted the latest updates to the webshell-xpages-ext-lib to include components that support browser websockets on Domino/XWork server.  They are available on OpenNTF, and can be installed independently from the rest of the project.  The Domino plugin / component leverages the efforts from the Java-Websocket project (a bare bones websocket api).  Using their API made it surprisingly easy to spawn a server instance from the XPages run-time.  It is not complete, and could use some more enhancements (i.e. queue up a websocket message from a document, maybe a RESTful interface as well).  To try it out follow the instructions websocket-setup.pdf from the download.  The sample application is a simple chat app.  Please let me know how it goes, or if you find any bugs.

    Thursday, January 09, 2014

    Using the XPages Toolbox, Guice, & Method Interceptors to profile Java methods

    In 2013, I was fortunate enough to attend a few master class sessions at IBM Connect.  During one of those master class sessions, I was introduced to the XPages Toolbox by speakers Tony Mcguckin, and Marie Kehoe (two very talented engineers at IBM).  Among other things, the XPages toolbox provides statistics on method invocation times for SSJS and Java (with a little work).  By default, all SSJS methods are registered with the profiler which makes it relatively simple to find any performance bottlenecks.  Java, on the other hand, does require some custom code modifications to capture profiling data into the toolbox.  Below is a code sample from the XPages Toolbox documentation outlining how to use it in java code:

    private static final ProfilerType profilerCustom = new ProfilerType("MyJavaType");
    public void myMethod() {
     if(Profiler.isEnabled()) {
      ProfilerAggregator agg=Profiler.startProfileBlock(profilerCustom, null); 
      long startProfiler=Profiler.getCurrentTime();
      try {
      } finally {
       Profiler.endProfileBlock(agg, startProfiler);
     } else {
    private void _myMethod() {
     try {
     } catch(InterruptedException ex){}

    As you can see from the above snippet, trying to profile many Java methods in your application would be very cumbersome, as you have to wrap each and every method implementation inside the profiler's start and end methods.  If you have a rough idea of what the performance problem could be, this might be an OK approach.   

    What to do if you are unsure where the code bottleneck is?  Do you pollute all your java methods with a bunch of profiler code?  Heck no... use Google's Guice (pronounced juice) and method interceptors.  Let's start with what the profiled code will look like after we implement a Guice method interceptor:

    public void myMethod() {
        try {
     } catch(InterruptedException ex){}

    Notice in the above snippet, we've only added a single annotation called @Profiled.  This alerts Guice that the method "myMethod" needs to be intercepted.  Nice, clean, and easy to implement and roll back once you have setup Guice.  As with most things there are some trade offs to this approach.  Below is a list of potential issues of using this method to profile your Java classes in the XPages runtime:

    • You have to use and get familiar with Guice.  The basics are pretty straight forward, but it may not align with how you or your organization currently develop.  I highly recommend it as it promotes an interface heavy design and makes swapping out implementations a snap.  As your applications / inter-dependencies become more complex, Guice helps to manage those dependencies without having to create tons of getter/setter methods.  More importantly, Guice limits the amount of code changes required to provide a new implementation.  There are plenty of blog entries on the benefits of using Guice, so google around.

    • If your Java classes reside inside your .nsf, the java.policy file must be altered and opened up completely on the server for Guice to work its injection/interceptor "magic".  This problem can be circumvented if you develop your core java classes and business logic outside the nsf as an extension library (I typically do this).

    • You must hand off all object creation to Guice for method interceptors to work (even injecting post construction won't work, though dependency injection does).  To have Guice create certain managed beans, use a variable resolver to instantiate the required objects via Guice (see GuiceVariableResolver in project in github).  Completely relying on Guice to create your objects is not so great as some of the nice scope storage mechanisms / behaviors are lost.  Once profiling is completed, you could always revert back to letting the XPages / JSF runtime create the instances by configuring your managed beans in faces-config, and then inject after construct (see GuiceVariableResolver).

    Below is a java class, and sample screen capture of the output from @Profiled annotated method in the LookupData managed bean (found in

    Simple Java Bean
    import java.util.ArrayList;
    import java.util.List;
    import javax.faces.model.SelectItem;
    public class LookupData implements ILookupData {
     public List getAccidentTypes(){
      List list = new ArrayList(2);
      list.add(new SelectItem("PD","property damage"));
      list.add(new SelectItem("PI","personal injury"));
      return list;
     public List getSeverities(){
      List list = new ArrayList(2);
      list.add(new SelectItem(1,"low"));
      list.add(new SelectItem(2,"medium"));
      list.add(new SelectItem(3,"high"));
      return list;

    Output from XPages Toolbox

    To disable the method intercept altogether, just comment out the ProfilerModule when applying the Guice modules, and the @Profiled annotation will be ignored (see AirtsActivator in  You could also just leave the interceptor in place and disable the XPages profiler in the notes.ini.  The interceptor will still get called, but will not attempt to log the method invocation times.

     public static List getModules(){
      List modules = new ArrayList();
      modules.add(new DominoModule());
      modules.add(new ManagedBeanModule());
      //modules.add(new ProfilerModule());
      return modules;

    You can grab the sample airts application from github in the below repo:

    The sub-projects to look into and relevant classes are:

      • AirtsActivator (inits Guice)
      • GuiceVariableResolver (creates guice objects and/or injects dependencies)
      • ILookupData, and LookupData (simple java bean class)
      • ManagedBeanModule (shows the Guice bindings, and defines what vars are created by guice)
      • Profiled (annotation class for the method interceptor)
      • ProfileInterceptor (intercepts the method and logs the profiling data)
      • ProfileModule (sets up the Guice bindings, and the method interceptor)
      • IGuicer (Guicer interface)
      • Guicer (wrapper for Guice injector)

    Quick Install Notes:
    If you receive a security exception when attempting to use this code, drop the following line into the default grant statement of the java.policy file on your domino server:

    permission org.osgi.framework.AdminPermission "*", "metadata";

    The above will open up access to the metadata associated with the plugins on your server, which is required by the plugin.

    If you're trying to profile java code that is inside your own .nsf and you're getting security exceptions, backup the java.policy file, then replace the contents of the default grant statement with the below.  Think about moving your java code outside of the .nsf as an ext-lib, as that environment has more execution rights than code executed from within the .nsf.  You won't have to open up the permissions like this...


    Thanks for taking the time.