Orchestrating Microservices with Conductor: A Developer's Guide

conductor-oss/conductor

At its core, Conductor is a workflow orchestration platform. Think of it as a central nervous system for your microservices. In a typical microservice architecture, you have many small, independent services that need to work together to complete a business process. Conductor helps you manage the flow and order of these services.

The key benefit here is that it decouples the logic of your application's business process from the services themselves. Instead of services having hardcoded knowledge of their dependencies, Conductor defines the workflow as a separate, declarative model. This makes your services more reusable and your overall system much more resilient. It's especially useful in scenarios where a complex process involves multiple steps, retries, and error handling.

From a software engineer's perspective, Conductor addresses several common pain points in building distributed systems

Reliable and Durable Workflows
Conductor ensures that even if a service fails, the overall workflow doesn't just crash. It can retry tasks, handle failures gracefully, and persist the state of the workflow so it can be resumed later. This is crucial for long-running processes like order fulfillment or data processing.

Simplified Error Handling
Instead of writing complex, boilerplate code for retries and compensation logic in each service, you can define these rules directly in the Conductor workflow. This makes your code cleaner and your error handling more consistent.

Improved Observability
Conductor provides a dashboard and APIs to monitor the status of every running workflow. You can see which tasks are completed, which are running, and where a failure occurred. This is a massive help for debugging and troubleshooting.

Scalability
Since Conductor is built for distributed environments, it can handle a large number of concurrent workflows and tasks. It's designed to be highly available and scalable to meet the demands of modern applications.

Business Process Visualization
The declarative workflow model makes it easy to visualize and understand complex business processes. You can literally see a diagram of the steps, which is great for both engineering teams and business stakeholders.

Let's walk through a simple example using a hypothetical "User Onboarding" workflow. This process might involve creating a user account, sending a welcome email, and then provisioning their profile.

First, you define your workflow in a JSON or JavaScript file. This is where you specify the tasks and their order.

user_onboarding_workflow.json

{
  "name": "user_onboarding_workflow",
  "version": 1,
  "description": "Orchestrates the user onboarding process",
  "tasks": [
    {
      "name": "create_user_account",
      "taskReferenceName": "create_user_account",
      "type": "SIMPLE"
    },
    {
      "name": "send_welcome_email",
      "taskReferenceName": "send_welcome_email",
      "type": "SIMPLE",
      "inputParameters": {
        "email_address": "${create_user_account.output.email}"
      }
    },
    {
      "name": "provision_user_profile",
      "taskReferenceName": "provision_user_profile",
      "type": "SIMPLE"
    }
  ]
}

In this example, ${create_user_account.output.email} shows how you can pass data between tasks.

Next, you create your "workers" (microservices) that will perform the actual work. These workers poll Conductor for tasks they are registered to handle.

JavaScript Example (create_user_account.js)

const { WorkflowClient } = require('@orkes-io/conductor-javascript');

const client = new WorkflowClient({
  serverUrl: 'http://localhost:8080/api', // Your Conductor server URL
});

client.startWorker('create_user_account', async (input) => {
  console.log('Received task to create user account:', input);
  
  // Simulate creating a user in a database
  const user = {
    userId: 'user-123',
    email: input.email,
    username: 'john.doe'
  };

  console.log(`User created: ${user.userId}`);
  
  return {
    status: 'COMPLETED',
    outputData: {
      userId: user.userId,
      email: user.email
    }
  };
});

You would create similar workers for send_welcome_email and provision_user_profile.

Finally, you can start a new workflow execution by making a simple API call to the Conductor server.

Example using Node.js/JavaScript

const { WorkflowClient } = require('@orkes-io/conductor-javascript');

const client = new WorkflowClient({
  serverUrl: 'http://localhost:8080/api',
});

async function startOnboarding() {
  const workflowId = await client.startWorkflow({
    name: 'user_onboarding_workflow',
    version: 1,
    input: {
      email: '[email protected]'
    }
  });

  console.log(`Started new workflow with ID: ${workflowId}`);
}

startOnboarding();