Developed an Apache Lucene component in Camel to perform indexed searches on a route

I have just submitted a new Lucene Component to the Apache Camel community that

• builds a searchable index of documents when payloads are sent to the Lucene Endpoint
• facilitates performing indexed searches in Camel when the payload header contains a QUERY.

http://issues.apache.org/activemq/browse/CAMEL-1472

The component works as follows

Creating a Searchable Document Index in Lucene using Camel

context.addRoutes(new RouteBuilder() {
    public void configure() {
        from("direct:start")
            .to("lucene:stdQuotesIndex:insert?"
               + "analyzer=#stdAnalyzer"
               + "&indexDir=#std&srcDir=#load_dir")
            .to("mock:result");

    }
});

where each URI parameter setting does the following

• analyzer: can be any valid implementation of Lucene Directory Analyzer (StandardAnalyzer, WhitespaceAnalyzer, StopAnalyzer... etc)
• srcDir: an optional directory location for loading Text or XML documents at endpoint or Lucene Index creation. Once created the index can take any exchange body and store its contents in the index.

Important Note: Lucene stipulates that the index be created upfront and then used in a read only mode later for any querying. Hence the index cannot be in flux during query processing. This requires the Lucene Producer to have received its payloads upfront and created the index before any queries can be logged against it.

Since the URI settings cannot be directly passed (since they are object references or break the URI format), I pass them using the JNDI registry associated with the the Default Component (example shown below).

Providing URI values for Analyzer and Initial Load Directory

@Override
protected JndiRegistry createRegistry() 
   throws Exception {
   JndiRegistry registry =
       new JndiRegistry(createJndiContext());
   registry.bind("std", new File("target/stdindexDir"));
   registry.bind("load_dir",
       new File("src/test/resources/sources"));
   registry.bind("stdAnalyzer",
       new StandardAnalyzer(Version.LUCENE_CURRENT));

   return registry;
}

I have also added a QueryEndpoint and a Query Processor that is fully capable of running any queries (including wildcards etc) against a Lucene Document Index and present the results in a serialized Hits object (see example provided below for use)

Performing searches using a Query Endpoint

context.addRoutes(new RouteBuilder() {
   public void configure() {
            
     from("direct:start").
        setHeader("QUERY", constant("Seinfeld"))
        .to("lucene:searchIndex:query?"
            + "analyzer=#whitespaceAnalyzer"
            + "&indexDir=#whitespace"
            + "&maxHits=20")
        .to("direct:next");
            
     from("direct:next")
     .process(new Processor() {
        public void process(Exchange exchange)
           throws Exception {
           Hits hits = 
              exchange.getIn().getBody(Hits.class);
           printResults(hits);
        }

        private void printResults(Hits hits) {
           LOG.debug("Number of hits: " 
              + hits.getNumberOfHits());
           for (int i = 0; i < hits.getNumberOfHits(); i++) {
              LOG.debug("Hit " + i + " Index Location:" 
                 + hits.getHit().get(i).getHitLocation());
              LOG.debug("Hit " + i + " Score:"  
                 + hits.getHit().get(i).getScore());
              LOG.debug("Hit " + i + " Data:" 
                 + hits.getHit().get(i).getData());
        }
     }
     .to("mock:searchResult");
});

Developed a camel-cache component to store, perform lookups and listen for cache events on a camel route

I have recently submitted a Camel cache component which is slated for release in Apache Camel version 2.1 in Septermber, 09

http://issues.apache.org/activemq/browse/CAMEL-1868

The Camel Caching component has the following abilities

a> In Producer mode, the component provides the ability to direct payloads in exchanges to a stored in a pre-existing or created-on-demand Cache. The producer mode supports operations to ADD/UPDATE/DELETE/DELETEALL elements in a cache. (Examples goven below)

b> In Consumer mode, the component provides the ability to listen on a pre-existing or created-on-demand Cache using an event Listener and receive automatic notifications when any cache activity take place (i.e ADD/UPDATE/DELETE/DELETEALL). Upon such an activity taking place, an exchange containing header elements describing the operation and cachekey and a body containing the just added/updated payload is placed and sent. In case of a DELETEALL operation the body of the exchanage is not populated.

The cache itself may be created on demand or if a cache of that name already exists then it is simply utilized with its original settings. The URL itself may take the following form

Configuring the cache

from("cache://MyApplicationCache" +
          "?maxElementsInMemory=1000" + 
          "&memoryStoreEvictionPolicy=" +
              "MemoryStoreEvictionPolicy.LFU" +
          "&overflowToDisk=true" +
          "&eternal=true" +
          "&timeToLiveSeconds=300" + 
          "&timeToIdleSeconds=true" +
          "&diskPersistent=true" +
          "&diskExpiryThreadIntervalSeconds=300")

// Note that all the attributes of the above URL are 
// standard ehCache settings that may be set at 
// Cache creation.

Given below are examples of how to create/set routes:

Producer Example 1: Adding keys to the cache with a body received from direct:start

RouteBuilder builder = new RouteBuilder() {
    public void configure() {
     from("direct:start")
     .setHeader("CACHE_OPERATION", constant("ADD"))
     .setHeader("CACHE_KEY", constant("Ralph_Waldo_Emerson"))
     .to("cache://TestCache1")
   }
};

Producer Example 2: Updating existing keys in a cache with a body received from direct:start

RouteBuilder builder = new RouteBuilder() {
    public void configure() {
     from("direct:start")
     .setHeader("CACHE_OPERATION", constant("UPDATE"))
     .setHeader("CACHE_KEY", constant("Ralph_Waldo_Emerson"))
     .to("cache://TestCache1")
   }
};

Producer Example 3: Deleting existing keys in a cache with a body received from direct:start

RouteBuilder builder = new RouteBuilder() {
    public void configure() {
     from("direct:start")
     .setHeader("CACHE_OPERATION", constant("DELETE"))
     .setHeader("CACHE_KEY", constant("Ralph_Waldo_Emerson"))
     .to("cache://TestCache1")
   }
};

Producer Example 4: Deleting all keys in a cache with a body received from direct:start

RouteBuilder builder = new RouteBuilder() {
    public void configure() {
     from("direct:start")
     .setHeader("CACHE_OPERATION", constant("ADD"))
     .setHeader("CACHE_KEY", constant("Ralph_Waldo_Emerson"))
     .to("cache://TestCache1");

     from("direct:start")
     .setHeader("CACHE_OPERATION", constant("ADD"))
     .setHeader("CACHE_KEY", constant("Ralph_Waldo_Emerson2"))
     .to("cache://TestCache1");

     from("direct:start")
     .setHeader("CACHE_OPERATION", constant("DELETEALL"))
     .to("cache://TestCache1");
    }
};

Consumer Example 1: Notifying any changes registering in a Cache to Processors and other Producers

// Note: in this example the consumer is 
//     created first and then 3 routes 
//     send different message as Cache Producers

RouteBuilder builder = new RouteBuilder() {
    public void configure() {
     from("cache://TestCache1")
     .process(new Processor() {
        public void process(Exchange exchange) 
               throws Exception {
           String operation = (String) exchange.getIn().getHeader("CACHE_OPERATION");
           String key = (String) exchange.getIn().getHeader("CACHE_KEY");
           Object body = exchange.getIn().getBody();
           // Do something
        } 
     })
   } 
};

I have added a set of nice processors to the camel-cache component to provide the ability to perform cache lookups and selectively replace payload content at the
- body
- token
- xpath level

The mechanics of doing this are as follows

Producer Example 3: Deleting existing keys in a cache with a body received from direct:start

RouteBuilder builder = new RouteBuilder() {
   public void configure() {
     //Message Body Replacer
     from("cache://TestCache1")
     .filter(header("CACHE_KEY").isEqualTo("greeting"))
     .process(new CacheBasedMessageBodyReplacer("cache://TestCache1", "farewell"))
     .to("direct:next");  

    //Message Token replacer
    from("cache://TestCache1")
    .filter(header("CACHE_KEY").isEqualTo("quote"))
    .process(new CacheBasedTokenReplacer("cache://TestCache1", "novel", "#novel#"))
    .process(new CacheBasedTokenReplacer("cache://TestCache1", "author", "#author#"))
    .process(new CacheBasedTokenReplacer("cache://TestCache1", "number", "#number#"))
    .to("direct:next");

    from("cache://TestCache1").
    .filter(header("CACHE_KEY").isEqualTo("XML_FRAGMENT"))
    .process(new CacheBasedXPathReplacer("cache://TestCache1", "book1", "/books/book1"))
    .process (new CacheBasedXPathReplacer("cache://TestCache1", "book2", "/books/book2"))
    .to("direct:next");
   }
};

Things to consider when selecting between Apache Camel and Apache Servicemix

Camel and Servicemix are both fundamentally about applying Enterprise Integration Patterns to solve complex integration problems across different technology sets.

However they do this by employing different styles and approaches. The broad distinctions between Camel and the Servicemix JBI technology stack, in my view, are the following

• Camel is standardized via DSL (Domain Specific Language), Servicemix on the other hand is an implementation of the JBI standard

• Both use message exchanges internally to move payloads between endpoints using routes (Camel) and workflows (JBI). However, in Camel, a MessageExchange is an Interface which is implemented by different technology sets in their own way i.e HTTP Exchange, FTP Exchange etc and employs type conversion to normalize data between these message exchanges. In Servicemix, a Message Exchange is a standardized Data structure with XML as its message payload.

• Messages in Camel may be of any type (binary, text, serialized object etc). Messages in Servicemix are XML only.

• Camel utilizes URI's to express endpoints, Servicemix uses the WSDL conventions (ServiceEndpointReference) to express endpoints.

• Camel has a broader palette due to a simpler/more efficient and expressive way of wiring endpoints using routes. It is more easily embeddable in traditional Java Apps and can be run from a command line. Servicemix on the other hand is fundamentally based on a framework (JBI) and its services/endpoints are deployment in a JBI/OSGi based container.

• The core JBI framework provides its endpoints key infrastructure services/facilities out of the box (location transarency, mediated messaging via NMR) and is more suited for viewing as a traditional ESB. Camel is more suited for being viewed as an easy to use, yet sophisticated piece of DSL based integration technology.

BTW, Servicemix 4 (OSGi) blends/brings together both these technologies beautifully. Servicemix-Camel and Camel can seamlessly coexist and provide seamless navigation between JBI and Camel routes.

I like both for their own quirks, elegance & oddities. To each his own, I guess :).

Using Camel to direct messages to a Printer using a camel-printer component

I have recently made a feature submission of a camel-printer component to the Apache Camel community. I believe it will be available in Apache Camel 2.1 release cycle.

You can follow the submission to the Apache community at the following JIRA link

https://issues.apache.org/activemq/browse/CAMEL-1675

The objective of this component is to provide a way to direct payloads on a route to a printer. Obviously the payload has to be a formatted piece of payload in order for the component to appropriately print it. The objective is to be able to direct specific payloads as jobs to a line printer in a camel flow.

The functionality allows for the payload to be printed on a default, local remote or wirelessly linked printer using the javax printing API under the covers.

Since the URI scheme for a printer has not been standardized (the nearest thing to a standard being the IETF print standard) and therefore not uniformly applied by vendors, I have chosen "lpr" as the scheme.

There is choice for setting the following options in the printer URI

Printer options

- mediasize: to set the stationary as defined by 
           enumeration settings in the 
           javax.print.attribute.standard.MediaSizeName API.
- copies: to set number of copies based on the 
           javax.print.attribute.standard.Copies API
- sides: to set one sided or two sided printing based on the 
           javax.print.attribute.standard.Sides API

The way this camel printer works in Camel is as follows

Example 1: Printing text based payloads on a Default printer using letter stationary and one-sided mode

RouteBuilder builder = new RouteBuilder() {
    public void configure() {
     from(file://inputdir/?delete=true)
     .to("lpr://localhost/default?copies=2" +
         "&flavor=DocFlavor.INPUT_STREAM&" +
         "&mimeType=AUTOSENSE" +
         "&mediaSize=na-letter" +
         "&sides=one-sided")
   }
};

Example 2: Printing GIF based payloads on a Remote printer using A4 stationary and one-sided mode

RouteBuilder builder = new RouteBuilder() {
    public void configure() {
     from(file://inputdir/?delete=true)
     .to("lpr://remotehost/sales/salesprinter" +
         "?copies=2&sides=one-sided" +
         "&mimeType=GIF&mediaSize=iso-a4" +
         "&flavor=DocFlavor.INPUT_STREAM")
   }
};

Example 3: Printing JPEG based payloads on a Remote printer using Japanese Postcard stationary and one-sided mode

RouteBuilder builder = new RouteBuilder() {
    public void configure() {
     from(file://inputdir/?delete=true)
     .to("lpr://remotehost/sales/salesprinter" +
         "?copies=2&sides=one-sided" +
         "&mimeType=JPEG" +
         "&mediaSize=japanese-postcard" +
         "&flavor=DocFlavor.INPUT_STREAM")
   }
};

I hope you like this feature and use it to direct formatted payloads to a printer as part of a camel flow.

A Camel Dataformat that facilitates symmetric key encryption/decryption of XML payloads at the Payload, XML Element and XML Element Content level

I just developed and submitted a new Dataformat for Camel that facilitates encryption and decryption of XML payloads at the Document, Element and Element Content levels (including simultaneous multi-node encryption using XPATH).

The encrytion capability is based on formats supported using the Apache XML Security (Santuario) project. Encryption/Decryption is "currently" supported using Triple-DES and AES (128, 192 and 256) encryption formats. Additional formats can be easily added later as needed. (Note: The support currently offered is for symmetric encryption. This means the same keyset is needed at both ends of the communication to encrypt/decrypt payloads).

The capability allows Camel users to encrypt/decrypt payloads while being dispatched or received along a route.

The default encrytion format if no algorithm is specified is Triple-DES.

The way it works is as follows

Example 1: Full Payload encryption/decryption

RouteBuilder builder = new RouteBuilder() {
      public void configure() {
          from("http:www.foo.com/orders")
          .marshal().encryptXML()
          .unmarshal().encryptXML()
          .to("activemq:queue:ORDERS")

      }
};

Example 2: Partial Payload Content Only encryption/decryption

RouteBuilder builder = new RouteBuilder() {
      public void configure() {
          String tagXPATH="//cheesesites/italy/cheese";
          boolean secureTagContent = true;
      
          from("http:www.foo.com/orders")
          .marshal().encryptXML(tagXPATH, secureTagContent)
          .unmarshal().encryptXML(tagXPATH, secureTagContent)
          .to("activemq:queue:ORDERS")
      }
};

Example 3: Partial Multi Node Payload Content Only encryption/decryption

RouteBuilder builder = new RouteBuilder() {
      public void configure() {
          String tagXPATH = "//cheesesites/*/cheese";
          boolean secureTagContent = true;
          
          from("http:www.foo.com/orders")
          .marshal().encryptXML(tagXPATH, secureTagContent)
          .unmarshal().encryptXML(tagXPATH, secureTagContent)
          .to("activemq:queue:ORDERS")
      }
};

Example 4: Partial Payload Content Only encryption/decryption using passPhrase(password)

RouteBuilder builder = new RouteBuilder() {
      public void configure() {
          String tagXPATH = "//cheesesites/*/cheese";
          boolean secureTagContent = true;
          byte[] passPhrase = "Just another 24 Byte key".getBytes();
      
          from("http:www.foo.com/orders")
          .marshal().encryptXML(tagXPATH, secureTagContent, passPhrase)
          .unmarshal()
          .encryptXML(tagXPATH, secureTagContent, passPhrase)
          .to("activemq:queue:ORDERS")
      }
};

Example 5: Payload encryption/decryption using passPhrase with passPhrase Algorithm

RouteBuilder builder = new RouteBuilder() {
      public void configure() {
          String tagXPATH = "//cheesesites/*/cheese";
          boolean secureTagContent = true;
          byte[] passPhrase = "Just another 24 Byte key".getBytes();
          String algorithm= XMLCipher.TRIPLEDES;
      
          from("http:www.foo.com/orders")
          .marshal().encryptXML(tagXPATH , secureTagContent, passPhrase, algorithm)
          .unmarshal().encryptXML(tagXPATH, secureTagContent, passPhrase, algorithm)
          .to("activemq:queue:ORDERS")
      }
};

The other choices for algorithm are

--> XMLCipher.AES_128
--> XMLCipher.AES_192 and
--> XMLCipher.AES_256

For more details on the submission check out the following

https://issues.apache.org/activemq/browse/CAMEL-1360

A Message Compressor/Decompressor for messages sent using Camel routes

There is tremendous value in message compression of XML based payloads as they are dispatched to remote endpoints utilizing Camel as a mediation router.

Camel supports marshalling and unmarshalling of messages into and out of several different data formats such as JAXB, JSON, HL7 etc. An ability to map a message from XML into a compressed data format such as (ZLIB, GZIP, PKZIP) etc can offer significant advantages with respect to network performance and storage in exchange for a small added cost of message compression.

The messages marshalled using zip compression can be unmarshalled using zip decompression just prior to being consumed at the endpoint. The true value of applying compression during a camel route is that it requires no changes to the source or destination application. Compression takes place just after the message leaves the source application and is picked up by the Camel message listener. The message exchange can now be forwarded/routed via several endpoints in a camel flow. Ideally, the message need be uncompressed only when it arrives at its final destination.

I have implemented this feature/capability and made a submission to the Camel community. I believe this capability is invaluable when you deal with large XML and Text based payloads.

The way this feature is designed to work in Camel is as follows

Example 1: Message Compression

RouteBuilder builder = new RouteBuilder() {
    public void configure() {
        from("http:www.foo.com/orders")
        .marshal().zip(Deflater.BEST_COMPRESSION)
        .to("activemq:queue:ORDERS")
    }
};

Example 1: Message Decompression

RouteBuilder builder = new RouteBuilder() {
      public void configure() {
           from("activemq:queue:ORDERS")
           .unmarshal().zip()
           .bean((MyOrderProcessor.class);
      }
};

I hope you like this feature and use it judiciously in an enterprise environment that utilizes Camel as a mediation layer to route messages to their ultimate destinations as a compact set of payloads.

Please find my submission to the Apache community here

https://issues.apache.org/activemq/browse/CAMEL-1133

Easing development in Servicemix 4 using Archetypes

I recently submitted maven archetypes to the Apache Servicemix community for adding ease of use in Servicemix 4 in the following areas

• An OSGi based Camel maven archetype to ease development of Camel routes for deployment in the Servicemix 4 OSGi container (servicemix-osgi-camel-service-unit)
• A CXF WSDL first maven archetype to ease development of WSDL first web services (servicemix-osgi-cxf-wsdl-first-service-unit)
• An CXF Code first archetype to ease development of Code first web services (servicemix-osgi-cxf-code-first-service-unit)

If you are interested in using these archetypes check out the following jira issues at the Apache Servicemix community site. These fixes will be committed into the upcoming release of Servicemix 4.

• https://issues.apache.org/activemq/browse/SMX4-150
• https://issues.apache.org/activemq/browse/SMX4-142
• https://issues.apache.org/activemq/browse/SMX4-141

Opening up to Open Source

Over the last several years, I have had the opportunity to work on projects utilizing IONA supported versions of Open Source software from the Apache Foundation. I have delivered projects using

• FUSE ESB (Apache Servicemix)
• FUSE Message Broker (Apache ActiveMQ)
• FUSE Services Framework (Apache CXF) and
• FUSE Mediation Router (Apache Camel)

I have also had the good fortune to work with Eclipse Project chairpersons such as Eric Newcomer and Apache Committers, Thought Leaders and Project chairpersons such as Guillaume Nodet, James Strachan, Bruce Snyder and Dan Kulp.

I have also benefited from the knowledge of my "most excellent" consulting colleagues such as Rich Bonneau, Dhiraj Bokde, Adrian Trenaman, Rohan Mars, Peter Mackinnon, David Greco and many others.

I also developed the Open Source Courseware material for the technology sets mentioned above and delivered customer training, solution workshops and open enrolment classes.

Over the coming weeks, I plan on putting forward articles, use-cases, how-to's and other bits of information to hopefully ease the challenges in applying the above mentioned open source offerings in real world applications.

I hope you find these articles useful and applicable in your environments.