GitLab Self-hosted Agent Setup Checklist

This guide will help you set up a GitLab self-hosted agent for MiddlewareHQ.

Prerequisites

Before proceeding, ensure you have the following:

• Docker installed on your system and docker daemon is running.
• Middleware Credentials: Contact your Middleware Systems administrator to obtain the MHQ_CLIENT_ID and MHQ_CLIENT_SECRET credentials required for authentication.
  • Instructions for how to use them are below.
• Personal Access Token (PAT): Generate a PAT with the read-api and read-user scopes to authenticate with the GitLab API.

Setup Instructions

• Clone Configurations
  1. Clone this repository: https://github.com/middlewarehq/sync-agent-conf
• Middleware Credentials:
  1. Obtain the Middleware Credentials (MHQ_CLIENT_ID and MHQ_CLIENT_SECRET) from your Middleware Systems administrator.
  2. Add these to client.txt.
• Generate Personal Access Token (PAT):
  1. Obtain the domain URL (GITLAB_DOMAIN_URL) of your self-hosted GitLab instance in the format https://gitlab.com/-/user_settings/personal_access_tokens OR https://yourgitlabinstance.com/-/user\_settings/personal\_access\_tokens .
  2. Generate a PAT with the read_api and read_user scopes to authenticate with the GitLab API.
  3. Use this token (GITLAB_ACCESS_TOKEN) for authentication and update it when necessary for security reasons.
  4. Add both to client.txt.
• Define Repository URLs:
  1. Save the repository URLs in config.json for reference during synchronization in, integrations->gitlab->repositories
• Deploy the Agent:
  1. Run the following command to create and run the Docker container for the MHQ GitLab Agent in the project root:
     docker compose up -d
     This command deploys the agent in detached mode, allowing it to run independently in the background.
     (If you’re running this later, run docker compose pull)

Additional Checks

1. Check if the docker container is running properly:

   1. docker ps -a
      Use the above command to get the list of all the docker containers. Copy the id of the docker-container with the name “mhq-sync-scripts”.

      1. You may also try this command:
         docker ps -a | grep mhq-sync-scripts | awk '{print $1}'

   2. docker logs -f [id_of_container]
      Use the above command with the id of the container you copied from the previous step. This command will give the logs of the container.
      For convenience, try this:
      docker logs -f $(docker ps -a | grep mhq-sync-scripts | awk '{print $1}')

   3. Check the logs to match the following. The order of the logs may differ.
      You want to ensure you see the following (regardless of their order):

      1. data-extraction-and-transfer entered RUNNING state
      2. python-heartbeat entered RUNNING state
      3. cron entered RUNNING state
      4. If you see “FATAL state” for any of the above, get in touch with us.

   4. Finally, the exit status of data-extraction-and-transfer shall be 0 (shows up as something like data-extraction-and-transfer (exit status 0; expected)). It may take some time to get this last log as it depends on the amount of data it is trying to sync.

   5. Now go to check #2 to see your data on Middleware!
      If you don’t see any data, try the next point 👇

   6. If data not found in Middleware, run the following commands:

      1. Copy the container ID using:
         docker ps -a | grep mhq-sync-scripts | awk '{print $1}'

      2. docker exec -it [id_of_mhq_scripts_container]

      3. python3 data_extraction.py

      4. Once it completes, run
         python3 data_transfer.py

      5. After that, exit the container simply by running
         exit

      6. Continue to to check #2 👇

2. Check Middleware App Integration
   Visit your Middleware App (app.middlewarehq.com) to check if your app has integrated with the GitLab self-hosted agent successfully

3. Go to https://app.middlewarehq.com/integrations

4. Gitlab’s integration should appear linked. The “Link” button as shown below would turn into “Unlink”
   

   1. If it worked, you should see this:
      
   2. If you don’t see “Unlink” here, try the steps outlined here 👉 (link), or contact us.

5. Visit https://app.middlewarehq.com/teams

6. Select a team of your choice from the list

7. Click on the “Repos” tab right under the page header, see image
   

8. Your repos will show up under “Repos linked to your org”

9. Click on the << icon on each repo you want to add to the team
   

10. A “notification” will indicate that your settings have been saved

11. That’s it!

    1. Process Overview (https://app.middlewarehq.com/collaborate/process) is a great place to start viewing your first insights.
    2. If you don’t see anything even after 5 minutes or so, let us know on jayant@middlewarehq.com or dhruv@middlewarehq.com.

12. Database Management (optional):
    Use a database management tool like TablePlus to connect to the Docker PostgreSQL database and inspect logs for further analysis.
    Ensure to configure with the following parameters:

• Host: localhost or server IP address

• Port: 5435

• User: postgres

• Password: postgres

• Database: mhqagent

  NOTES:

1. Password change instructions coming soon.
2. We recommend not exposing port 5435 on the machine that this is running on to any external network.

Follow above steps to successfully set up and configure the GitLab self-hosted agent for MiddlewareHQ synchronization.