Latest News

…copy files with SwitchYard

Description

While copying files from one place to another we sometimes face performance problems, and as the file gets bigger more problems, even OutofMemoryExceptions. These problems can be related to how we transfer files in SwitchYard, and if we do conversions of the files.
Here there is an explanation of the different ways to transfer a file, in streaming mode, or as a String.

Project

This project demonstrates how to copy files as stream or as a String.
  • I have used ESB interfaces, an used java.io.InputStream as input param for the copy files as stream.
  • I have used ESB interfaces, an used java.lang.String as input param for the copy files as string.
You can manage where to do conversion by having the left side as InputStream and right side as String, so you can convert at the ServiceHandler, at the Exchange (implicit transformers).
In the attached project drop a file called test.txt in /input. It will get copied in Streaming mode to /output. In the attached project drop a file called string.txt in /input. It will get copied in String mode to /output.
If using big files, copy the file with another name, and then mv it to the correct name, as otherwise copy will be happening not for the
If modifying the source, adapt source and destinations folders and filenames

Results

1 KB message

As Stream:
13:50:23,222 DEBUG [org.apache.camel.component.file.FileConsumer] (Camel (camel-15) thread #15 - file:///input) Total 1 files to consume
13:50:23,222 DEBUG [org.apache.camel.component.file.FileConsumer] (Camel (camel-15) thread #15 - file:///input) About to process file: GenericFile[/input/test.txt] using exchange: Exchange[test.txt]
13:50:23,224 DEBUG [org.apache.camel.component.file.GenericFileConverter] (Camel (camel-15) thread #15 - file:///input) Read file /input/test.txt (no charset)
13:50:23,231 DEBUG [org.apache.camel.component.file.FileOperations] (Camel (camel-15) thread #15 - file:///input) Using InputStream to write file: /output/ID-b64fd2f46bfd-46028-1412943213101-14-60
13:50:23,232 DEBUG [org.apache.camel.component.file.GenericFileProducer] (Camel (camel-15) thread #15 - file:///input) Wrote [/output/ID-b64fd2f46bfd-46028-1412943213101-14-60] to [Endpoint[file:///output]]
13:50:23,234 DEBUG [org.apache.camel.component.file.GenericFileOnCompletion] (Camel (camel-15) thread #15 - file:///input) Done processing file: GenericFile[/input/test.txt] using exchange: Exchange[null]
13:50:23,275 DEBUG [org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy] (Camel (camel-15) thread #15 - file:///input) Renaming file:
GenericFile[/input/test.txt] to: GenericFile[/input/.camel/test.txt]
As String:
13:50:46,976 DEBUG [org.apache.camel.component.file.FileConsumer] (Camel (camel-15) thread #16 - file:///input) Total 1 files to consume
13:50:46,976 DEBUG [org.apache.camel.component.file.FileConsumer] (Camel (camel-15) thread #16 - file:///input) About to process file: GenericFile[/input/string.txt] using exchange: Exchange[string.txt]
13:50:46,977 DEBUG [org.apache.camel.component.file.GenericFileConverter] (Camel (camel-15) thread #16 - file:///input) Read file /input/string.txt (no charset)
13:50:46,983 DEBUG [org.apache.camel.component.file.FileOperations] (Camel (camel-15) thread #16 - file:///input) Using InputStream to write file: /output/ID-b64fd2f46bfd-46028-1412943213101-14-70
13:50:46,984 DEBUG [org.apache.camel.component.file.GenericFileProducer] (Camel (camel-15) thread #16 - file:///input) Wrote [/output/ID-b64fd2f46bfd-46028-1412943213101-14-70] to [Endpoint[file:///output]]
13:50:46,986 DEBUG [org.apache.camel.component.file.GenericFileOnCompletion] (Camel (camel-15) thread #16 - file:///input) Done processing file: GenericFile[/input/string.txt] using exchange: Exchange[null]
13:50:46,987 DEBUG [org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy] (Camel (camel-15) thread #16 - file:///input) Renaming file:
GenericFile[/input/string.txt] to: GenericFile[/input/.camel/string.txt]
There is no significative difference.

400 MB file

As Stream:
13:52:58,301 DEBUG [org.apache.camel.component.file.FileConsumer] (Camel (camel-15) thread #15 - file:///input) Total 1 files to consume
13:52:58,302 DEBUG [org.apache.camel.component.file.FileConsumer] (Camel (camel-15) thread #15 - file:///input) About to process file: GenericFile[/input/test.txt] using exchange: Exchange[test.txt]
13:52:58,304 DEBUG [org.apache.camel.component.file.GenericFileConverter] (Camel (camel-15) thread #15 - file:///input) Read file /input/test.txt (no charset)
13:52:58,311 DEBUG [org.apache.camel.component.file.FileOperations] (Camel (camel-15) thread #15 - file:///input) Using InputStream to write file: /output/ID-b64fd2f46bfd-46028-1412943213101-14-80
13:52:58,526 DEBUG [org.apache.camel.component.file.GenericFileProducer] (Camel (camel-15) thread #15 - file:///input) Wrote [/output/ID-b64fd2f46bfd-46028-1412943213101-14-80] to [Endpoint[file:///output]]
13:52:58,528 DEBUG [org.apache.camel.component.file.GenericFileOnCompletion] (Camel (camel-15) thread #15 - file:///input) Done processing file: GenericFile[/input/test.txt] using exchange: Exchange[null]
13:52:58,528 DEBUG [org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy] (Camel (camel-15) thread #15 - file:///input) Renaming file:
GenericFile[/input/test.txt] to: GenericFile[/input/.camel/test.txt]
As String:
13:53:42,017 DEBUG [org.apache.camel.component.file.FileConsumer] (Camel (camel-15) thread #16 - file:///input) Total 1 files to consume
13:53:42,017 DEBUG [org.apache.camel.component.file.FileConsumer] (Camel (camel-15) thread #16 - file:///input) About to process file: GenericFile[/input/string.txt] using exchange: Exchange[string.txt]
13:53:42,019 DEBUG [org.apache.camel.component.file.GenericFileConverter] (Camel (camel-15) thread #16 - file:///input) Read file /input/string.txt (no charset)
13:53:43,535 DEBUG [org.apache.camel.component.file.FileConsumer] (Camel (camel-15) thread #15 - file:///input) Took 0.000 seconds to poll: /input
13:53:48,535 DEBUG [org.apache.camel.component.file.FileConsumer] (Camel (camel-15) thread #15 - file:///input) Took 0.000 seconds to poll: /input
13:53:51,794 ERROR [org.apache.camel.processor.DefaultErrorHandler] (Camel (camel-15) thread #16 - file:///input) Failed delivery for (MessageId: ID-b64fd2f46bfd-46028-1412943213101-14-83 on ExchangeId: ID-b64fd2f46bfd-46028-1412943213101-14-84). Exhausted after delivery attempt: 1 caught: org.apache.camel.TypeConversionException: Error during type conversion from type: org.apache.camel.component.file.GenericFile to the required type: java.lang.String with value GenericFile[/input/string.txt] due java.lang.OutOfMemoryError: Java heap space: org.apache.camel.TypeConversionException: Error during type conversion from type: org.apache.camel.component.file.GenericFile to the required type: java.lang.String with value GenericFile[/input/string.txt] due java.lang.OutOfMemoryError: Java heap space
at org.apache.camel.impl.converter.BaseTypeConverterRegistry.convertTo(BaseTypeConverterRegistry.java:126) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:72) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:47) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.switchyard.component.camel.common.composer.CamelMessageComposer.compose(CamelMessageComposer.java:62) [switchyard-component-common-camel-1.1.1-p7-redhat-1.jar:1.1.1-p7-redhat-1]
at org.switchyard.component.camel.common.composer.CamelMessageComposer.compose(CamelMessageComposer.java:42) [switchyard-component-common-camel-1.1.1-p7-redhat-1.jar:1.1.1-p7-redhat-1]
at org.switchyard.component.camel.SwitchYardProducer.composeForGateway(SwitchYardProducer.java:136) [switchyard-component-camel-switchyard-1.1.1-p7-redhat-1.jar:1.1.1-p7-redhat-1]
at org.switchyard.component.camel.SwitchYardProducer.process(SwitchYardProducer.java:104) [switchyard-component-camel-switchyard-1.1.1-p7-redhat-1.jar:1.1.1-p7-redhat-1]
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:122) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:298) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:117) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.fabric.FabricTraceProcessor.process(FabricTraceProcessor.java:81) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:334) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:220) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:46) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:308) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:117) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:46) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:336) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:189) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:155) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:142) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:92) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_65]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [rt.jar:1.7.0_65]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [rt.jar:1.7.0_65]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65]
Caused by: org.apache.camel.RuntimeCamelException: java.lang.OutOfMemoryError: Java heap space
at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1326) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:972) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:47) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:253) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
at org.apache.camel.impl.converter.BaseTypeConverterRegistry.convertTo(BaseTypeConverterRegistry.java:111) [camel-core-2.10.0.redhat-60061.jar:2.10.0.redhat-60061]
... 57 more
Caused by: java.lang.OutOfMemoryError: Java heap space

13:53:51,796 DEBUG [org.apache.camel.component.file.GenericFileOnCompletion] (Camel (camel-15) thread #16 - file:///input) Done processing file: GenericFile[/input/string.txt] using exchange: Exchange[null]
13:53:51,796 WARN [org.apache.camel.component.file.GenericFileOnCompletion] (Camel (camel-15) thread #16 - file:///input) Rollback file strategy: org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@164d2158 for file: GenericFile[/input/string.txt]
As you can see, streaming is recommended for big files. So be careful whatever you do in an Streaming file copy with the message content, as any log, transformation,…​ can convert it to String and give an OOME.

…SwitchYard development guidelines (I)

This document describes guidelines and best practices on how to develop services. Although there are many ways of achieving the same functionality in SwitchYard, these is the recommended approach to ease development effort and to leverage all the capabilities provided by the engine.

Strongly type your contracts

As the input type in an operation’s contract is the Message, you should always use a type if you expect a message. Even if it works without a message it is in no way recommended, and can lead to multiple problems with difficult dianostic.

Wrong

public interface MyWrongContract {
public void operation();
}

Right

public interface MyRightContract {
public void operation(MyType input);
}

Use the appropiate types in your contracts

If you want to leverage the type conversion provided with SwitchYard, and also be able to efficiently work with your information (objects/types) in your components it is strongly adviced to use the appropiate types, and not relying on generic types, that will need conversions.
Sometimes can be convenient to use a generic type, mostly to avoid conversions, but it is not a general rule.

Wrong

public interface MyContract {
public void operation(org.w3c.dom.Element input);
}

Right

public interface MyContract {
public void operation(com.company.type.MyType input);
}

Use declarative transformations

When you have to tranform a type, like when a Composite Service interface is different from the Component Service interface, you can instruct SwitchYard to do the transformation for you. It will be applied automatically while processing the wire.
You can add required transformers or a specific tranformation, and you can use one of the multiple transformations mechanisms. See: https://docs.jboss.org/author/display/SWITCHYARD/Transformation

If using Java transformations, create common functionality, to not repeat yourself

When using Java transformations, and more if transforming to/from XML, you’ll end up adding same code in many transformations. Extract a base class into your utils to ease Java/XML conversion.
public abstract class BaseTransformers {

/**
* Gets an element value for transforming from Element to Java Type
*
* Usage: MyType myTransform(Element e) {
* String s = getElementValue(e, "xxx");
* return new MyType(s);
* }
*/

private String getElementValue(Element parent, String elementName) {
String value = null;
NodeList nodes = parent.getElementsByTagName(elementName);
if (nodes.getLength() > 0) {
value = nodes.item(0).getChildNodes().item(0).getNodeValue();
}
return value;
}

/**
* Creates an Element, for tranforming from Java Type to Element
*
* Usage: public Element myTransform(MyType from) {
* String xml = "<xxx>" + from.toString + "</xxx>";
* return toElement(xml);
* }
*
*/

private Element toElement(String xml) {
DOMResult dom = new DOMResult();
try {
TransformerFactory.newInstance().newTransformer().transform(new StreamSource(new StringReader(xml)), dom);
} catch (Exception e) {
e.printStackTrace();
}
return ((Document)dom.getNode()).getDocumentElement();
}

Use @OperationTypes in your interface if you want to instruct SwitchYard for a different type than that in the interface

Sometimes it can be convenient to have the XML representation of an Object and not require SwitchYard to do a transformation from the XML type to a String. For this cases (and many others) can be convenient to annotate your interface so SwitchYard knows it doesn’t need to transform to the String type.
public interface OrderService {

@OperationTypes(in = "{urn:switchyard-quickstart:bean-service:1.0}submitOrder")
void submitOrder(String orderXML);
}
In this way, the String that will get to the submitOrder method will be:
<urn:submitOrder xmlns:urn="urn:switchyard-quickstart:bean-service:1.0">
<order>
<orderId>1</orderId>
<itemId>1</itemId>
<quantity>100</quantity>
</order>
</urn:submitOrder>

Get access to the Context in a Transformer

@Context annotation is limited to be used inside of components. If you need to get access to the context in a SwitchYard transformer, you can qualify the “from” and use Message as input type in the transformer method.
@Transformer(from = "java:com.example.ClassOne")
public ClassTwo transform(Message message) {
Context context = message.getContext();
....
return classTwo;
}

Use MessageComposers to adapt the message transformation from/to Message from/to Binding

Message composers are used to translate the binding message (SOAPEnvelope, HTTP Body, Rest Body, File contents, …​) into the SwitchYard Message. This is the place to put logic to customize this transformation. And only transformation of the message/body.
Create a custom MessageComposer (best to extend the one for the binding you are using) and override the appropriate method:
compose
Takes the data from the passed in source object and composes a SwithYardMessage based on the specified Exchange.
public Message compose(D source, Exchange exchange) throws Exception;
decompose
Takes the data from the SwitchYardMessage in the specified Exchange and decomposes it into the target object.
public D decompose(Exchange exchange, D target) throws Exception;

Use ContextMappers to customize context mapping from/to Context from/to Binding

Context mappers are used to translate the binding context (SOAP Headers, HTTP Headers, …​) into the SwitchYard Context. This is the place to put logic to customize the creation and assigment of information into the Context.
Create a custom ContextMapper (best to extend the one for the binding you are using) and override the appropriate method:
mapFrom
Maps a source object’s properties to the context.
public void mapFrom(D source, Context context) throws Exception;
mapTo
Maps a context’s properties into a target object.
public void mapTo(Context context, D target) throws Exception;

Use SCA to decompose and decouple your applications

Usually applications start to grow big, and very coupled the information in there. Many wires from services to components to wires,…​ In such an scenario it is very good option to think as SCA binding as the solution, as introduces 2 benefits:
  • Allows for easy decouplig and composition of applications.
  • Application dependency is only honoured at runtime. You can deploy appA that has an sca call to appB before even deploying appB. It will only fail at runtime if you do not deploy appB.

Externalize configuration from the system (but be aware of the limitations)

Applications should not have configuration information that may be different in different environments hard coded. Even component, composite and domain properties are not good for this, as they require rebuilding of the application, so use JBossAS properties or System.properties or environment to externalize your configuration.
Configuration is applied typically at activation of a service, so any Environment Property will be read, and if changed will require, at a minimum, from an application reactivation, or a server restart.
JBossAS7 properties can be changed from the console, and if you read them in your execution path will be hot-reloaded, otherwise, if it is configuration property, will still require an application reactivation.
static property
This property will be read once (when the class is instantiated) and needs an application reactivation/redeployment to change
static String password = System.getProperty("password");

public void process(Exchange exchange) throws Exception {
....
}
dynamic property
This property will be read per every invocation and will be changed in real time
public void process(Exchange exchange) throws Exception {
String password = System.getProperty("password");

....
}

..SwitchYard ExchangeHandler Chains

Communications between components and bindings to/from components is done by a LocalExchangeBus.
This is the wire
There is a chain of ExchangeHandlers acting as interceptors for the request and response pipelines of the wire. The default chain has the following:
  • RequestChain
    • addressing: AddressingHandler resolves service instances based on a service reference.
    • transaction-pre-invoke: TransactionHandler interprets transactional policy specified on an exchange and handles transactional requirements. Suspends current transaction and creates a new one, creates a new one, join an ongoing transaction,…​
    • security-process: SecurityHandler interprets security policy specified on an exchange and handles security requirements. Cretes the SecurityContext and SecurityProvider and attaches to the thread.
    • generic-policy: PolicyHandler
    • validation-before-transform: ValidateHandler executes the org.switchyard.validate.Validator defined for the Exchange`s content type.
    • transformation: TransformHandler applies all the defined transformations (org.switchyard.transform.TRansformer) for the message for consumer.inputType to provider.inputType and set the Exchange’s new content type.
    • validation-after-transform: ValidateHandler executes the org.switchyard.validate.Validator defined for the new Exchange`s content type.
    • provider: ProviderHandler dispatches the message to the provider (component).
    • security-cleanup: SecurityHandler cleanup of context and thread info.
    • transaction-post-invoke: TransactionHandler commits or rollback, or resume suspended transaction,…​
Only IN_OUT Exchanges will execute the ReplyChain, that’s why there are 2 Handlers after the execution of the provider, to cleanup/finish tasks.
  • ReplyChain
    • validation-before-transform: ValidateHandler executes the org.switchyard.validate.Validator defined for the Exchange`s content type.
    • transformation: TransformHandler applies all the defined transformations (org.switchyard.transform.TRansformer) for the message for provider.outputType to consumer.outputType and set the Exchange’s new content type.
    • validation-after-transform: ValidateHandler executes the org.switchyard.validate.Validator defined for the new Exchange`s content type.
    • HandlerChain.CONSUMER_HANDLER: BaseHandler

… SwitchYard ServiceHandlers

Every SwitchYard application has a composite service with one or multiple bindings.

Every binding acts as an endpoint. Incoming endpoint if it is a binding on a service or outgoing endpoint if it is a binding on a reference.
This endpoints are created, registered and started when the aplication gets deployed, and are stoped, unregistered and destroyed when the application is undeployed.
Every endpoint implements org.switchyard.deploy.ServiceHandler
There are inbound handlers and outbound handlers, depending if they act as server endpoints or client endpoints.
Inbound handlers:
  • org.switchyard.component.camel.common.handler.InboundHandler: An handler that acts as a gateway/entrypoint for Camel Components.
  • org.switchyard.component.http.InboundHandler: Hanldes HTTP requests to invoke a SwitchYard service.
  • org.switchyard.component.resteasy.InboundHandler: Handles RESTEasy requests to invoke a SwitchYard service.
  • org.switchyard.component.soap.InboundHandler: Handles SOAP requests to invoke a SwitchYard service.
  • org.switchyard.component.sca.SCAEndpoint: For publishing services as SCA
Outbound handlers:
  • org.switchyard.component.camel.common.handler.OutboundHandler: A handler that is capable of calling Apache Camel components.
  • org.switchyard.component.http.OutboundHandler: Handles invoking external HTTP services.
  • org.switchyard.component.resteasy.OutboundHandler: Handles invoking external RESTEasy services.
  • org.switchyard.component.soap.OutboundHandler: Handles invoking external Webservice endpoints.
  • org.switchyard.component.sca.SCAInvoker: Handles outbound communication to an SCA service endpoint

How does an InboundHandler Works

When it is created, an instance of the BindingModel (configuration) is passed, as well as an instance of the ServiceDomain, is passed to the InboundHandler.
When it starts creates the ServiceReference representing the endpoint, the endpoint for the corresponding binding and the message composer.
Provides one method, invoke, where all the logic for an invocation to the endpoint happens.
When it stops, the endpoint is stopped.

InboundHandler.invoke()

  • creates an Exchange
  • add identy to the Exchange to identify the origin of the invocation
  • compose the message (MessageComposer.compose()) from the passed binding Message (BindingData)
  • send the Message to the Exchange (Exchange.send())
  • If it is IN_OUT:
    • it will wait for a response (Default timeout of 5 minutes)
    • it will decompose the response (MessageComposer.decompose())
  • it returns the decomposed message as BindingData

How does a message translation works

Here we will find a description of what is happening at the message composition time in a call to a SwitchYard application through an external binding and having one call to an external reference through a binding.
  1. Request message from binding to service (Request from client). The request message is converted from the binding message type to the internal SwitchYard Message, and all the binding headers (HTTP, SOAP, …​) will be mapped into the corresponding properties in the SwitchYard Context associated to the Message, and it is set to the Exchange.
  2. Service component implementations execution. The request pipelines of a component will be executed, until a there is a call to an external service.
  3. Request message from reference to binding (External service call). A call to the external service is about to happen, and message needs to be converted from SwitchYard Message to the reference binding data, the MessageComposer will decompose the message back to the binding format and properties will be extracted from the SwitchYard Context and set into the binding message.
  4. Response message from binding to reference (Response from remote service). Once the external service has been executed, the response from this external service will be mapped back into SwitchYard by the MessageComposer.mapFrom(). The ContextMapper will copy the appropiate message/transport headers back into the Message Context
  5. Service execution. The rest of the components` pipeline will be executed.
  6. Response message from service to binding (Response to client caller). When our application has ended it’s execution and it is returning the result to the caller, the MessageComposer will decompose the SwitchYard message back to the binding message and the ContextMapper will map the properties in the Context back to the binding message.
Additional behaviour relative to the correspoding transaltions will exist in the predefined MessageComposers and ContextMappers (SOAP, REST, HTTP,…​)

How does an OutboundHandler Works

When it is created, an instance of the BindingModel (configuration) is passed, as well as an instance of the ServiceDomain, is passed to the OutboundHandler.
When it starts creates and configures a ENdpoint consumer to access the external service through the corresponding Binding method (HTTP, SOAP, REST, …​).
Provides one method, handleMessage, where all the logic for an invocation to the external service happens.
No stop lifecycle.

OutboundHandler.handleMessage()

  • add identy to the Exchange to identify the origin of the invocation
  • decompose the message (MessageComposer.decompose()) to a binding Message (BindingData)
  • it will use the dispatcher (HTTP, SOAP, REST, …​) to invoke the real service with the binding data
  • If it is IN_OUT:
    • it will wait for a response
    • it will compose the response (MessageComposer.compose()) from the received BindingData back into a Message
  • send the Message to the Exchange (Exchange.send()) (or Exchange.sendFault() if it was an error)

… SwitchYard Context

Introduction

Holds contextual information around the exchange of messages between a service consumer and provider. SwitchYard context resembles Camel Context and in the execution of a Camel Component delegates/extends CamelContext.
It is just a placeholder for name/value pairs, but internally holds Property objects consisting of a name, scope, value and labels.
  • Interface definition is org.switchyard.Context.
  • Default implementation is org.switchyard.internal.CompositeContext
Context has scopes (Message and Exchange) and the life of this contexts is bound to the life of the scope it defines.
  • A property defined with Scope.MESSAGE will live as long as the Message that defined the property lives.
  • A property defined with Scope:EXCHANGE will live for the entire life of the Exchange, surviving every message that is sent to the Exchange.
You can get a property from the context in a specific scope with getProperty(String name, String scope) or regardless of the scope with getProperty(String name)
You can add a property, label a property, remove a property in one scope, in any scope, all the properties,…​

Labels

Labels are a way to mark a Context Property, and relate it to a SwitchYard funtcionality.
Base class is org.switchyard.label.Label
There are 5 types of labels (OOTB):
  • BehaviorLabel: Marks the property relative to certain actions and/or behavior. (Copy of context,…​) (label type:behavior)
    • TRANSIENT: the property will not be preserved in a copy of the Exchange via Context.copy()
  • ComponentLabel: Marks the property relative to certain component (label type: component)
    • CAMEL, HORNETQ, HTTP, JCA, RESTEASY, SOAP
  • EnpointLabel: Marks the property relative to Endpoint related actions (context composition, message composers,…​) (label type: endpoint)
    • AMQP, ATOM, DIRECT, FILE, FTP, FTPS, HTTP, JCA, JMS, JPA, MAIL, MOCK, QUARTZ, REST, SEDA, SFTP, SOAP, SQL, TCP, TIMER, UDP, URI
  • HttpInvokerLabel: Marks the property to be used as a header in the HTTPInvoker (label type: httpinvoker)
    • HEADER
  • PropertyLabel: Marks the type of the property. It is used in JMS to mark JMS Properties or JMS Headers (label type:property).
    • HEADER, PROPERTY

SwitchYard labels

A label is composed following this rule:
prefix.<lowercase(label_type)>.<lowercase(label_name)>
  • prefix: “org.switchyard.label.”
  • label type: (behavior, component, endpoint, httpinvoker, property)
  • label name: (TRANSIENT, CAMEL, HTTP, JCA,…​.., HEADER, PROPERTY)
Examples:
org.switchyard.label.behavior.transient
org.switchyard.label.endpoint.http
org.switchyard.label.endpoint.soap
org.switchyard.label.component.soap
org.switchyard.label.property.header

Usage

SwitchYard uses the context for 3 main purposes:
  • Storing internal information for the engine to work (defining behavior, lifecycle, …​). Every SwitchYard defined property is named with org.switchyard. prefix
  • Binding components store information in the Context (camel components mainly)
  • User specific storage of information.

SwitchYard usage of the Context

SwitchYard stores information in the context, in the appropiate Scope to model its behavior. It uses labels to mark the context of the information in the context, and to be able to remove the properties from the context when appropriate. Some of the functionalities in the Context and Labels are not meant to be used by users, only by SwitchYard internals, although they are defined in the same place.

User usage of the Context

User is subject to store information in the Context, also to model the application behavior in terms of user functionality. A user can use the Context in every SwitchYard component, in the MessageComposers and ContextMappers, transformers,…​ but how to access the context varies.

Components

Java
You can Inject the Context in any Java Component (Service Implementation) by using the @Component annotation
Camel
SwitchYard context properties are mapped to the Exchange properties and Message headers based on the scope (Scope.MESSAGE, Scope.EXCHANGE). There is no need to interact with the SwitchYard context in a Camel component.
  • SwitchYard Exchange Properties are mapped to Camel Properties.
 from("xxx")
.setProperty("MyHeader", "MyValue")
.to("yyy");
  • SwitchYard Message Properties are mapped to Camel Message Headers.
 from("xxx")
.setHeader("MyHeader", "MyValue")
.to("yyy");
BPMN/Rules
There are some Implicit Variables when defining the “from” in mappings from SY to Component.
  • context: The current org.switchyard.Context.
  • message: The current org.switchyard.Message.
Some examples:
from="message.content" - This is the same as message.getContent().
from="context[‘foo’]" - This is the same as context.getProperty("foo", Scope.MESSAGE).getValue(), in a null-safe manner.
BPEL
TODO

MessageComposers

In a MessageComposer you have the Exchange as a parameter to compose/decompose methods, so to get access to the context you can:
context = exchange.getContext();

ContextMappers

In a ContextMapper you have the Context as a parameter to mapFrom/mapTo methods.

Transformers

If you need to get access to the context in a SwitchYard transformer, you can qualify the “from” and use Message as input type in the transformer method.
@Transformer(from = "java:com.example.ClassOne")
public ClassTwo transform(Message message) {
Context context = message.getContext();
....
return classTwo;
}

ExchangeInterceptors

In ExchangeInterceptors you have the Context as a parameter to before/after methods.

How is the information stored in the Context

Once you have access to the Context, every Property looked up in the context (if looked up without explicitly passing a Scope) will be first looked up as Message scoped and the as Exchange scoped. If the user sets a property without explicitly targeting a Scope, this Property will be defined as Message scoped.
Some examples of Context usage:
Context context = exchange.getContext();
context.setProperty("BindingName", gatewayName, Scope.MESSAGE);
context.setProperty("org.switchyard.exchangeGatewayName"); // This property will be Message scoped

context.getProperty("org.switchyard.exchangeGatewayName", Scope.Exchange);
context.getProperty("BindingName"); // Will return Message scoped property if defined, or then Exchange scoped

context.getProperties(); // Will provide with a Set<Property> with every property in every context
context.getProperties(Scope.EXCHANGE); // Will provide with a Set<Property> with every property in EXCHANGE context

context.removeProperties(); // Clears out the context
context.removeProperties(Scope.MESSAGE); // Clears out the MESSAGE Scope context
context.removeProperties(Label); // Removes every property in every context with that Label

Exchange context

This is a graphical representation of the Exchange context
Exchange Context ->
CamelBatchComplete .............................: true
CamelBatchIndex ................................: 0
CamelBatchSize .................................: 1
org.switchyard.bus.camel.consumer ..............: ServiceReference [name={....}]
org.switchyard.bus.camel.contract ..............: org.switchyard.metadata.BaseExchangeContract@49cb9565
org.switchyard.bus.camel.dispatcher ............: org.switchyard.bus.camel.ExchangeDispatcher@65994f93
org.switchyard.bus.camel.labels ................: {CamelFileExchangeFile=[org.switchyard.label.component.camel, org.switchyard.label.endpoint.file],...., org.switchyard.security.context.SecurityContext=[org.switchyard.label.behavior.transient]}
org.switchyard.bus.camel.phase .................: IN
org.switchyard.bus.camel.replyHandler ..........: org.switchyard.component.camel.CamelResponseHandler@54a37b1a
org.switchyard.exchangeGatewayName .............: file1
org.switchyard.exchangeInitiatedNS .............: 10999671294181171
org.switchyard.security.context.SecurityContext : DefaultSecurityContext@1210402593[systemUUID=455e975a-de16-4450-ac34-4b358fe1915d, expirationMillis=0, credentials=[], securityDomainsToSubjects={}]

Message Context

This is a graphical representation of the Message context
Exchange Context ->
Message Context ->
BindingName .........................: file1
CamelFileAbsolute ...................: true
CamelFileAbsolutePath ...............: /input
CamelFileLastModified ...............: 1412686922000
CamelFileLength .....................: 702
org.switchyard.bus.camel.labels .....: {CamelFileAbsolutePath=[org.switchyard.label.component.camel, org.switchyard.label.endpoint.file], ...., CamelFileLength=[org.switchyard.label.component.camel, org.switchyard.label.endpoint.file]}
org.switchyard.bus.camel.messageSent : true
org.switchyard.messageComposer ......: com.xxx.MyMessageComposer@1a6a1e9f
timeIn ..............................: 20141007150250407

User defined labels

A user might want to create a label to mark and to add/remove information that they add in an easy way. It is just a matter of defining a new subclass of Label, and add the enumeration of labels needed.
In this way, a user can add information into the context, and then remove all of their information at once:
context.addProperty("userTransactionId","x12fd25", Scope.EXCHANGE, UserDefined.USER.label());
...
context.removeProperties(UserDefined.USER.label());

… message composers in SwitchYard

Message composition

It is the process of converting the Message from the binding format (HTTP request, SOAP envelope, JMS message, File, …​) to a SwitchYard message, and mapping the context information from the binding format (HTTP headers, SOAP headers, JMS headers, …​) to the SwitchYard Context, and from the Switchyard internal format (message and context) back again into the binding format.

Message Composition Explained with an example

Here we will find a description of what is happening at the message composition time in a call to a SwitchYard application through an HTTP binding and having one call to an external reference through an HTTP binding.

(1) Request message from binding to service

The request message is converted from the binding message type to the internal service type. For this to happen, a call toHttpMessageComposer.compose().
In this method two things will happen:
  • It will create a Message for the Exchange.
  • A call to HttpContextMapper.mapFrom(HttpRequestBindingData, context) will be done. This method will:
    • It will copy requestInfo from the HTTPRequestBindingData into the message Context with keyHttpComposition.HTTP_REQUEST_INFO and assigning the following labels, ComponentLabel.HTTP andEndpointLabel.HTTP
HttpRequestInfo has all the required information from the request, and is the source of information that can be used in the SwitchYard’s component implementations to access information relative to the request, such as requestURI, requestPath, method, contextPath,…​ See the implementation of this class to find all the available information.
  • It will copy all headers from the HTTP request that match the defined includes and exludes regex pattern for the composer at design time.
Headers are not automatically copied, you have to add to messageComposer which headers you want to include (.*: all)
When checking for headers, or defining headers in Binding’s RegEx Inclusion and Exclusion, all headers arelowercased. If you send a header Authorization, you should check for authorization instead.
  • It will copy the content from the HttpBindingData to the Message that will flow through SwitchYard converting it to the appropiate type.
  • It will return the Message.

(2) Service component implementations execution

The request pipelines of a component will be executed, until a there is a call to an external HTTP service.

(3) Request message from reference to binding

A call to the external service is about to happen, and message needs to be converted from SwitchYard message to the reference binding data.
  • A call to HttpMessageComposer.decompose(Exchange, HttpRequestBindingData) will be done. This method will:
    • Copies the content from the Exchange to the content HttpRequestBindingData
    • Calls HttpContextMapper.mapTo(Request) to copy headers from the Message`s Context to the HttpRequestBindingData’s http headers. These properties will be copied only if they match the registered regexp expressions in the MessageComposer’s design time configuration or if the properties are marked with the EndpointLabel.HTTP label in the Context.

(4) Response message from binding to reference

Once the external service has being executed, we have to deal with the response from this external service, and map it back into SwitchYard. To to this, a call to *HttpMessageComposer.compose(HttpResponseBindingData, Exchange) will be done. This method will:
  • Create a Message
  • Call HttpContextMapper.mapFrom(HttpResponseBindingData, Message) that will:
    • Copies the statusCode of the invocation into a Context property calledHttpContextMapper.HTTP_RESPONSE_STATUS with labels http (ComponentLabel.HTTP and EnpointLabel.HTTP)
    • It will copy all headers from the HTTP response that match the defined includes and exludes regex pattern for the composer at design time.
Headers are not automatically copied, you have to add to messageComposer which headers you want to include (.*: all)
  • It will copy the content from the HttpBindingData to the Message that will flow through SwitchYard converting it to the appropiate type.
  • It will return the Message.

(5) Service execution

The rest of the components` pipeline will be executed.

(6) Response message from service to binding

When our application has ended it’s execution and it is returning the result to the caller, it will call againHttpMessageComposer.decompose(Exchange, HttpResponseBindingData) to convert the SwitchYard message to the binding message or content. It will: * Get statusCode from Context propertyHttpContextMapper.HTTP_RESPONSE_STATUS that has been established as part of the SwitchYard service execution.
This property could be the one that was set from the invocation to the external Composite’s Reference. (In step 4)
  • If there is no statusCode set as a property, or this statusCode property does not have the EndpointLabel.HTTP label, it will calculate one depending on the content and on the Exchange.getState. If the state of the Exchange isExchangeState.FAULT it will return an HTTP Error code of 500. (See the code for the logic determining the statusCode)
    • Copy the content from the SwitchYard’s Message to the HttpResponseBindingData
    • Call HttpContextMapper.mapTo(HttpResponseBindingData) that will:
      • Copy headers from the Message`s Context to the HttpRequestBindingData’s http headers. These properties will be copied only if they match the registered regexp expressions in the MessageComposer’s design time configuration or if the properties are marked with the EndpointLabel.HTTP label in the Context.
      • If the property is HttpContextMapper.HTTP_RESPONSE_STATUS it will set the status on the HttpResponseBindingData

jBPM 6.1.0.Final available

The bits have been out for a few weeks now (as we wanted to wait on the new website for the public announcement), but we are now glad to announce that jBPM 6.1 is available and you can find all information about it on our new website.Some useful links:…

Uncategorized

jBPM6 Developer Guide

My twitter is exploding with references to this new book for a few days now, so I thought I’d share it with others as well: Mariano, Maurcio and Esteban have published a new version of their book a few weeks ago, jBPM6 Developer Guide.  Maria…