Skip to main content

Gitlab Self hosted Sync Agent

The GitLab Self-hosted Sync Agent is a tool designed to synchronize data between GitLab instances and MiddlewareHQ (MHQ) using a webhook-based integration. Here's a detailed explanation of its functionality and operation:

Purpose:​

  • Fetch data from GitLab instances.
  • Send fetched data to a webhook of MiddlewareHQ.
  • Provide a heartbeat system to notify MiddlewareHQ about the agent's health.

Functionality:​

  • Input: Customers provide authentication token and repository/project links they wish to sync with MiddlewareHQ.
  • Cron Job (Every 6 hours):
    • Fetch data from GitLab instances.
    • Process unnecessary data that doesn't need to be stored with MiddlewareHQ.
  • Cron Job (Every 6 hours):
    • Send processed data to MiddlewareHQ using a webhook.
  • Heartbeat System:
    • Periodically sends heartbeat to MiddlewareHQ to inform about the agent's health.

Minimum Architectural Requirements:​

  • Cloud Provider: AWS
  • Instance: t2.micro
  • CPU: 1 core
  • Memory: 1GB
  • Root Volume: 8GB
  • Data Storage: Postgres (In-memory, no extra setup required)

Operational Details:​

The agent can be deployed as a Docker container on an AWS EC2 instance.
It functions as a cron job scheduler, running every 6 hours to fetch and send data.
The agent utilizes two self-created Docker images:

  1. mhq-sync-scripts: Contains scripts for syncing data between GitLab and MiddlewareHQ. https://gallery.ecr.aws/y4x5l0o7/mhq-sync-scripts
  2. dbmate-docker: Provides a Dockerized Postgres instance for data storage. https://gallery.ecr.aws/y4x5l0o7/dbmate-docker

Configuration:​

  • Customers need to configure the agent with GitLab authentication token and repository/project links.
  • MiddlewareHQ webhook URL needs to be provided for data transmission.
  • Further the user needs to get a public encryption key from their Middleware Admin to allow data transmission.
  • AWS EC2 instance needs to be provisioned with at least minimum resources defined above.
  • Postgres database is included in the Docker image and doesn't require additional setup.

Documentation:​

Detailed setup instructions are available in the Gitlab Self-hosted Checklist document.
Docker images can be found in the provided AWS ECR repository links.