Crossfire Engine Multi-threading

Overview

This page outlines how to setup the core Crossfire Engine steps to run independently of each other.

Crossfire has 5 primary steps in it's processing life-cycle

    1. Decompile

    2. Process

    3. Compile

    4. Send

    5. UpdateSent

By default, the Crossfire Engine runs each of these steps and they apply to all of the messages going through the Crossfire environment. E.g. if you had 100 messages in a 'Decompiled' status, when the 'Process' step runs, then it will process all of those messages, regardless of which partner, or entity they are for.

These steps can be separated by Partner, or Entity, and in certain cases, Compile Groups (to be explained below). When this is done, it effectively creates totally separate processing threads for each partner, meaning a high volume of messages from one partner will not affect another partner.

In affect you end up with separate life-cycles, e.g.

Lifecycle 1

    1. Decompile - Customer 1

    2. Process - Customer 1

    3. Compile - Customer 1

    4. Send - Customer 1

    5. UpdateSent - Customer 1

Lifecycle 2

    1. Decompile - Customer 2

    2. Process - Customer 2

    3. Compile - Customer 2

    4. Send - Customer 2

    5. UpdateSent - Customer 2

If Customer 1 receives 1000000 messages, Customer 2's messages will be largely unaffected and continue to process normally (aside from load on other shared areas, e.g. database)

Configuration - Per Partner or Per Entity Processing Life-cycles

The life cycles can be separated by Entity or Partner. Specifying a specific Entity or Partner can be done in the Schedule Step configuration.

A standard configuration would like like this

If you were to specify a specific Entity, this would be done using a configuration like this, where the number on the end is the EntityID.

If you were to specify a specific Partner, this would be done using a configuration like the one below, where the value on the end is the partner code.

All of the steps (Decompile, Process, Compile, Send, UpdateSent) can have a 'Partner' or 'Entity' specified on them. The step will then only run for those Partners or Entities.

If you do not specify an Entity or Partner, then it will run for ALL entities, including those which have their own steps. E.g. If you have two schedules.

Schedule 1: "Decompile"

Schedule 2: "Decompile:Partner:MAX"

Both schedules will decompile interchanges for the partner with the code 'MAX', but Schedule 2 will ONLY decompile interchanges

Configuration - Multi-threads for any Partner or Entity

Most of the life-cycle processing steps in Crossfire can be configured to run on multiple threads. E.g. You can configure multiple schedules exactly the same and each schedule will perform the same action on different messages, effectively increasing throughput by the order of the number of schedules your create - 2 schedules, typically 2 times the performance.

This doesn't work for the Compile step. This is because compile can collect multiple messages together, so must run on a single thread. Note: If two compile threads are set up exactly the same it may result that 1 messages is compiled twice and creates two outbound interchanges.

To handle this issue above, a compile group can be created. These can be made up and put on the Process, as shown below (highlighted in yellow).

Note: Each compile group, must only have a single schedule.

Note: Once a compile group is set on one process, it must be set on all processes. Every process must have a compile group, or none

This is how you create a schedule for a compile group.

Choosing How To Setup Compile Groups

Typically customers have a few high volume customers which they want to separate. Let's say for example you have 20 partners. Two of those partners have very high volumes of messages. In this case you would create 3 Compile Groups

    1. Partner1

    2. Partner2

    3. OtherPartners

Then you would configure three schedules, which would have values such has:

Compile:CompileGroup:Partner1

Compile:CompileGroup:Partner2

Compile:CompileGroup:OtherPartners

In addition to this you would have at least three schedules for Decompile, Process, Send, UpdateSent. They would not have any Compile Group set on them.

First Implementation Of Any Multi Threading

The following steps would typically be followed for the initial setup:

    1. Stop the Crossfire Service

    2. Add the appropriate Compile Group to all Processes (Processes can have the same Compile Group) if you're working with Compile Groups.

    3. Create the required schedules

    4. Start the Crossfire Service