Durable Task Framework – Episode III: Coded Orchestrations

Coded Orchestrations

When working in integration space, we commonly deal with terms like Orchestration, Business Process Execution etc. For this purpose we use differed powerful products like Enterprise Service Busses, Process Execution Engines, etc.

Following picture shows one BPMN2 example, which graphically represents some business process.

Business people love such diagrams. They especially like the idea of painting such diagrams, compiling them and executing them. This gives a feeling you would an artist in a role of software architect. But in reality, there is a majority of cases, when this approach does not work. I spent lot of time in projects as developer or architect making such pictures running. Realistically majority of time I spent in fixing some configuration in those nice “pictures” then on creating execution logic. If you run in such problems DTF might be the right approach.
But first, business guys must accept (if applicable) the idea of coding of the business process. This sounds sometimes more expensive, but it doesn’t have to be. Today, high programming languages like C# are so efficient, that it is truly easier to read few lines of C# than to read few lines of XML or JSON.
If you are a developer with knowledge of .NET and C# they (you) will love DTF. With DTF you don’t get monitoring engine and fancy orchestration diagrams. But you have so called coded-orchestrations. Most important component in integration system is messaging and scheduler. DTF gives you scheduler by default and used messaging system is Service Bus (Azure Service Bus or Service Bus for Windows Server). All for free of charge. An additionally, you don’t need to learn anything about Service Bus, because DTF seats between your code and SB.

Typical DTF application is nothing else than representation of some workflow. For example, to express the BPMN diagram shown above in code, we would write something like:

Most Business People will probably agree that this is not that readable form as BPMN diagram. But, this is a 100% mathematically exact representation of the process. Diagrams make often illusion of simplicity. This is good to understand and to define the business process. But this hides incredibly the complexity. It looks like, after you constructed a diagram all you need is one student who will make mechanical work. This is a big mistake, because “things“ happen at lower level.
 

Hello DTF
 
Let’s implement now the orchestration form code shown above. Create project of any type which targets System32 platform (any Desktop kind of application). Then add the reference to NuGet package as shown at the next picture:

This will install the assembly called ‘DurableTask’, which implements all Durable Task Framework. DTF introduces two new artefacts:

·       Task Hub Client and

·       Task Hub Worker

Task hub client is implemented in class TaskHubClient and is responsible for starting of orchestrations, getting the state of orchestration, termination of orchestration etc. Agnostically it is similar to WCF-Proxy and HttpClient.

Task hub worker is a server part. Similarly to WCF ServiceHost, hub worker is used to register the orchestrations as service and to run listener. The worker is implemented in a class TaskHubWorker.

First task in typical DTF project is to implement orchestration. To do this, we should implement a class which derives from TaskOrchestration<Result,Input>.

Once the orchestration is started the method RunTask is invoked. Inside of this method orchestration logic should be implemented. Please note that this method is designed as async method. There are also few more virtual methods, which you can use, but these are out of scope of this article.

For now, it is important, that RunTask is the only method, which needs to be implemented.
Orchestration gets as input a generic instance (in this example of type Input) and returns result (in this example of type Result). Inside of this method all required business logic and rules can be implemented in pure C#.
One orchestration can be executed with any logic implemented in the body of RunTask method. But the true power of DTF is in idea of task orchestration. Because of this common orchestration code inside of RunTask method should orchestrate tasks and not implement any other logic instead of logic used to schedule tasks. 

Similarly to TaskOrchestration, DTF offers a class TaskActivity<Input, Result>.

Following code shows how to implement a task.

The task is a class which implements a class TaskActivity<TInput, TResult>.
That class provides an abstract method Execute. In contrast to RunTask method of the orchestration, method Execute is a blocking method. Next sample shows how to popup a dialog on the console. The user can enter the user name in dialog and the task will return it to orchestration. Task can implement basically anything. But be aware of the host. If your orchestration is running as a service, showing of modal dialogs is not very good idea. The real business task would look like shown in the next code snippet:

After implementation of task(s) and orchestration(s), you will have to implement a host code.
As already mentioned, hosting code can be any kind of desktop application (SCM Service, Outlook AddIn etc.). Hosting code has to initialize and start orchestrations as services.
Following code shows how to achieve that.

The code shown above does two things. First it initializes the hub. Before hub can be started, all required tasks have to be registered on hub. If the hub will run N orchestrations, all tasks of all N orchestrations must be registered before hub is started. Registration order of tasks and their reference to orchestration are not important.  
The registration is done by AdTaskActivities and AddTaskActivitiesFromInterface.
First one is used to register tasks, which belongs to an orchestration. The later one is used to register tasks, which implement some specific interface. The only difference is agnostic, or how you want to deal with your tasks. Traditionally tasks are atomic activities of an orchestration in the way how integration industry understand them. Because DTF is a coding and not a designer approach, we have much more possibilities.
For example, we can define interface with few operations and every operation can play a role of a task. This is very useful when some sequence of tasks has to be executed.

Following is the interface which defines a list of sequential tasks used internally in Microsoft Azure to migrate server farms.

    public interface IMigrationTasks

    {

        Task<bool> ExportSite(string id, string subscriptionId, Application application);

        Task<bool> ImportSite(string id, string subscriptionId, Application application);

        Task<bool> MigrateServerFarmConfig(string id, string subscriptionId, Application application);

        Task<bool> UpdateWebSiteHostName(string subscriptionId, Application application);

        Task<bool> WhitelistSubscription(string subscriptionId);

        Task<bool> CleanupPrivateStamp(string subscriptionId);

        Task<bool> EnableSubscription(string subscriptionId);

        Task<bool> DisableSubscription(string subscriptionId);

        Task<bool> UpdateTtl(string subscriptionId);

    }

The implementation of this interface is internally used by Microsoft and is not a part of Open Source. But the interface above gives you an idea how to implement such scenarios.
To run the task from interface you will have to use following code inside of RunTask method in your orchestration:

  taskHub.AddTaskActivitiesFromInterface<IMigrationTasks>(new MigrationTasks());

This single line of code will run sequentially all methods on that interface.

Under the hub

Initialization of the hub explained previously also performs the setup of required Service Bus queues and Table Storage tables. The picture below on left shows 3 required queues.

DTF internally uses 3 dispatchers for task scheduling. Orchestrator is the queue which orchestrate execution of orchestrations. Worker is the queue which orchestrates execution of tasks. Finally tracking queue is used for tracking of orchestration states. The picture above on right shows InstanceHistory00 table in the storage. This is the table which contains history events of orchestration. It is optionally used if storageConnectionString is specified by creating of the hub in following code snippet:

TaskHubWorker taskHub = new TaskHubWorker(taskHubName,
                                          servicebusConnectionString,
                                          storageConnectionString);

If storage connection string is not specified, no tracking information will be used. In that case there are no runtime restrictions, but you will not be able to get instance history data and instance state information. For some scenarios this is anyhow not required. Daenet has extended DTF to support provider based State Persistence. The upcoming version of DTF will provide injectable service similarly to OWIN middleware in ASP.NET.

But, as developer you don’t have to necessary know all details behind under the hub. When working with DTF you will never have to directly send or receive a message from Service Bus.
It is completely transparent to you. But it is good to know few details under the hub like queues and tables which are used. Sometimes you will have to deal with them especially during development time. Last but not least, it is also good to know that all communication with Service Bus queues uses session queues (ordered messaging) and compressed messages, which are serialized with JSON serializer.

Recap

With DTF Microsoft provides an easy option for building robust distributed and scalable services with build-in state persistence and program execution check points. All of scenarios described on the beginning of this article are very easy to implement with DTF.

Some Azure products like Mobile services, API Management or TFS Online use Durable Task Framework for automating of complex workflows. Right now DTF does not offer dashboard and orchestration designer. Daenet has already successfully implemented DTF in some traditional integration projects. Ten years ago we had to hardly deal with different protocols and products like BizTalk ware great help. But this time is over. We deal today with web services and problem of implementation of an adapter is mostly not an issue anymore. We had to invest in a past in development of Business Activity Monitor for DTF, but now we have a mobile dashboard for all devices which shows semantically aggregated business events generated from DTF framework. As long coding is not an option in you organization DTF is not the way to go. But if you have developers in the team, DTF is definitely a valid option for many integration scenarios.

References:
The framework binaries are available as a NuGet package at: https://www.nuget.org/packages/DurableTask

Git Project:
https://github.com/affandar/durabletask

A developer guide:
http://abhishekrlal.files.wordpress.com/2013/06/service-bus-durable-task-framework-developer-guide.docx