Skip to main content

Middleware Self hosted Sync Agent

The Middleware Self-hosted Sync Agent is a tool designed to synchronize data between your data providers (such as Gitlab, Bitbucket, etc.) and MiddlewareHQ Servers using a webhook-based integration. Here's a detailed explanation of its functionality and operation:

Purpose​

  • Fetch data from your data provider instances.
  • Send fetched data to a webhook of MiddlewareHQ.
  • Provide a heartbeat system to notify MiddlewareHQ about the agent's health.

Functionality​

  • Input: Customers provide authentication token and repository/project links they wish to sync with MiddlewareHQ.
  • Cron Job (Every 6 hours):
    • Fetch data from the data provider instances.
    • Process unnecessary data that doesn't need to be stored with MiddlewareHQ.
  • Cron Job (Every 6 hours):
    • Send processed data to MiddlewareHQ using a webhook.
  • Heartbeat System:
    • Periodically sends heartbeat to MiddlewareHQ to inform about the agent's health.

Minimum Architectural Requirements​

  • Cloud Provider: AWS
  • Instance: t2.micro
  • CPU: 1 core
  • Memory: 1GB
  • Root Volume: 8GB
  • Data Storage: Postgres (In-memory, no extra setup required)

Operational Details​

The agent can be deployed as a Docker container on an AWS EC2 instance.
It functions as a cron job scheduler, running every 6 hours to fetch and send data.
The agent utilizes two self-created Docker images:

  1. mhq-sync-scripts: Contains scripts for syncing data between your data provider and MiddlewareHQ. https://gallery.ecr.aws/y4x5l0o7/mhq-sync-scripts
  2. dbmate-docker: Provides a Dockerized Postgres instance for data storage. https://gallery.ecr.aws/y4x5l0o7/dbmate-docker

Configuration​

  • Customers need to configure the agent with provider (BitBucket/Gitlab/etc.) authentication token and repository/project links.
  • MiddlewareHQ webhook URL needs to be provided for data transmission.
  • Further the user needs to get a public encryption key from their Middleware Admin to allow data transmission.
  • AWS EC2 instance needs to be provisioned with at least minimum resources defined above.
  • Postgres database is included in the Docker image and doesn't require additional setup.

Documentation​

Detailed setup instructions are available in the subsequent provider-specific docs.