Enterprise Integration Orchestrator (EIO)

Enterprise Integration Orchestrator

Present day Problem (Integration of 2 Seperate Systems)

Scenario: Dell salesperson sells 100 laptops to HCL Technologies. salesperson uses CRM(salesforce) and ERP(netsuite)
Salesforce (CRM - Customer Relationship Management): Tool used by sales and marketing teams. It manages potential deals, sales pitches, client conversations, and emails.
NetSuite or Oracle (ERP - Enterprise Resource Planning): It tracks inventory, processes invoices, handles shipping/fulfillment, and records accounting data.
Step 1: The Sales Stage (Salesforce): A salesperson at Dell speaks with a client(HCL) who wants 50 laptops for $50,000 of Brand(Dell). The salesperson tracks this in Salesforce as an active Opportunity. After a few calls, the client(HCL) signs the contract.The salesperson changes the Salesforce status from Negotiation to Closed-Won (Opportunity Closure).
Now salesforce has to write API/email to send data to Netsuite
Step 2: Operations & Finance (NetSuite) Sales team at NetSuite creates a Sales Order containing: Customer details: Address, billing contact. Product line items: 50 laptops. Financial terms: $50,000, payment terms, etc. Payment link to be sent to customer and sales person. Once payment done, Laptops packed shipped and sent, all info provided to sales person for tracking.

Problems in this workflow: salesforce has to write API/email to send data to Netsuite and viceversa.

Solution iPaaS: Jitterbit

Enterprise Workflow Integrator(Jitterbit) performing data sync between 2 seperated platforms

Requirements

Functional
1. Workflow Management: Different system(AWS, GCP, Azure, Salesforce, SAP, Oracle ERP, ServiceNow, Kafka, Databases, SFTP, REST APIs, GraphQL, MQTT, Mainframes) produces events in different formats and with different SLAs orchestration engine should provide a single reliable control plane capable of coordinating millions of distributed tasks
Non Functional
No data loss, Exactly-once processing (where possible), Fault tolerance, Recovery after crashes, Cross-cloud execution, Scalability, Observability

HLD

Block Diagram

%%{init: {'themeVariables': {'fontSize': '18px'}, 'flowchart': {'nodeSpacing': 55, 'rankSpacing': 70, 'padding': 18}}}%%
flowchart LR

%% External Components

SF[Salesforce CRM]

GLB[Global Load Balancer]

RLB[Regional Load Balancer]

GLB --> RLB

SF --> GLB

%% Jitterbit Cloud

subgraph Jitterbit["Jitterbit Cloud (VPC/VNet)"]

APIGW[API Gateway]

AUTH[OAuth / Authentication]

ORCH[Workflow Orchestrator]

RULES[Business Rules Engine]

TRANS[Transformation Engine]

STATE[(Workflow State DB)]

META[(Metadata DB)]

KAFKA[(Kafka)]

DLQ[(Dead Letter Queue)]

W1[Worker 1]

W2[Worker 2]

W3[Worker 3]

MON[Monitoring & Logging]

RLB --> APIGW

APIGW --> AUTH

AUTH --> ORCH

ORCH --> RULES

ORCH --> TRANS

ORCH --> STATE

ORCH --> META

TRANS --> KAFKA

KAFKA --> W1
KAFKA --> W2
KAFKA --> W3

W1 --> MON
W2 --> MON
W3 --> MON

W1 --> DLQ
W2 --> DLQ
W3 --> DLQ

end

%% NetSuite

subgraph NetSuite["NetSuite Cloud"]

NS[NetSuite ERP]

end

W1 --> NS
W2 --> NS
W3 --> NS

NS --> GLB

Flow Diagram

%%{init: {'themeVariables': {'fontSize': '17px'}, 'sequence': {'actorFontSize': '17px', 'messageFontSize': '16px', 'noteFontSize': '15px', 'actorMargin': 70, 'messageMargin': 45}}}%%
sequenceDiagram
autonumber

actor SalesRep

box LightBlue Salesforce Cloud
participant SAS as AppServer
participant SDB as SF DB
end

box LightPink Jitterbit Cloud
participant AGW as API GW
participant Orc as Orchestrator
App Server
participant Rules
participant Transform
participant Cache as Cache
participant JDB as DB
participant Kafka
participant Workers as Worker
Pods
end 

box LightYellow NetSuite Cloud
participant NAS as Netsuite
App Server
participant NDB as DB
participant NKafka as Netsuite
Kafka
end

SalesRep->>SAS: Order 100 laptops
HTTP POST
{
customer=HCL,
orderQty=100,
quotedPrice=50000$
}
SAS->>SDB: INSERT INTO
SDB-->>SAS: ok

SAS->>AGW:HTTP POST
{
customer=HCL,
orderQty=100,
quotedPrice=50000$
}
Note over AGW: Authenticate Salesforce
Oauth/Okta IDP
 Get JWT

AGW->>Orc: HTTP POST
Orc->>Rules: Evaluate Workflow
Rules-->>Cache: Is rule present?
Cache->>Rules: Yes
Rules-->>Orc: Create Sales Order

Orc->>Transform: Map Salesforce → NetSuite
Note over Transform: Netsuite listen on gRPC
Endpoint /abc

Transform->>Cache: Is data present?
Cache-->>Transform: No
Transform->>JDB: INSERT INTO
JDB-->>Transform: done

Transform-->>Orc: NetSuite TYPE Payload
HTTP POST
{
customer=HCL,
orderQty=100,
quotedPrice=50000$
}

Orc->>Kafka: NetSuite Payload

Kafka->>Workers: Consume Task

Workers->>NAS: Create Sales Order
HTTP POST
{
customer=HCL,
orderQty=100,
quotedPrice=50000$
}
NAS->>NDB: Create order
NDB-->>NAS: Ok
NAS->>NKafka: topic="order_created",
"payload"
Note over NKafka: via NotificationSvc
Call Mail GW
Email GW
SMS GW(Twilio, Msg91)

NAS-->>Workers: SalesOrderID

Workers-->>Orc: Success

Orc-->>SAS: Update Opportunity (Synced)

Crashed Worker: If a worker crashes halfway, the orchestrator needs to resume from the last completed step. we can include a persistent Workflow State Store.


Orchestrator
↓
Workflow State DB
↓
Resume Workflow

Rules Engine: Sort of a state machine which tells what next action should take place


Rules Engine
↓
Opportunity Closed
↓
Create Sales Orde
↓
Update Customer
↓
Notify Finance

Dead Letter Queue: A queue where failed topics should be indicated to operators, if after 10 attempts delivery to Netsuite consumer fails, the topic would be pushed to this queue


Worker
↓
Retry
↓
Retry
↓
Retry
↓
DLQ

Monitoring: Add a separate Monitoring stack.


Workers
↓
Prometheus
↓
Grafana
↓
ELK
↓
OpenTelemetry

Detailed Design can be


                 Salesforce Cloud

             Salesforce REST APIs
                     │
             Global Load Balancer
                     │
             Regional Load Balancer
                     │
                 API Gateway
                     │
──────────────────────────────────────────
             Jitterbit Cloud
        Authentication Service
                     │
         Workflow Orchestrator
          │      │       │
          │      │       │
      Rules   StateDB  MetadataDB
          │
      Transformation
          │
        Kafka Queue
          │
  ┌───────┼─────────┐
  │       │         │
Worker1 Worker2 Worker3
  │       │         │
 Connector Runtime
          │
──────────────────────────────────────────
               NetSuite Cloud
          REST API
             │
      Customer
      Sales Orders
      Inventory
      Contacts