Middleware's Self-Hostable GitLab Sync Agent API calls

Supported Architecture

-> arm64 and amd64 machines

Data Extraction Cron

Runs at minute 0 past every 6th hour UTC.

API Calls during extraction

1. /personal_access_tokens/self
   1. Frequency: 1
2. /groups
   1. Frequency: 1
3. /groups/{group_path}/projects
   1. Frequency: 1 * count(groups) * (pages)
   2. Example: Consider you have 2 gitlab groups in your organization. The first group has 10 projects and the second group has 90 projects. The default page size in our implementation is 20, ie, we fetch 20 projects at once and the next 20 in the next, and so on. So for the first group we will make only 1 API call to fetch 10 projects. For the second group we make 5 API calls to fetch projects in batches of 20. So in total we make: 1 + 5 = 6 API calls
4. /projects/{project_id}/merge_requests
   1. Frequency: count(projects) * (pages)
   2. Example: Consider you have 100 projects from the above example, and each of those projects have 100 mrs in the last 90 days (we only fetch last 90 days data). For fetching 100 mrs for a single project we will make 100/20 = 5 requests. So to fetch merge requests for all 100 projects we will make 100*5 = 500 requests.
5. /projects/{project_id}/merge_requests/{merge_request_id}/commits
   1. Frequency: 1 * count(MR)
   2. Example: Following the above example of 100 projects with 100 merge requests each. So we will have 100*100 = 10000 api request in total to fetch all the commits of every PR.
6. /projects/{project_id}/merge_requests/{merge_request_id}/notes
   1. Frequency: 1 * count(MR)
   2. Example: Following the above example of 100 projects with 100 merge requests each. So we will have 100*100 = 10000 api request in total to fetch all the notes of every PR
7. /projects/{project_id}/merge_requests/{merge_request_id}/diff
   1. Frequency: 1 * count(MR)
   2. Example: Following the above example of 100 projects with 100 merge requests each. So we will have 100*100 = 10000 api request in total to fetch all the notes of every diff
8. /groups/{group_path}/members/all
   1. Frequency: 1 * count(Groups)
   2. Example: We have 2 groups. So we will make 2 API requests.
9. /projects/{project_id}/languages
   1. Frequency: 1 * count(Projects)
   2. Example: We have 100 projects. So we will make 100 API requests
10. /projects/{project_id}/repository/contributors
    1. Frequency: 1 * count(Projects)
    2. Example: We have 100 projects. So we will make 100 API requests

Total Requests made in 1 sync for an Organization with 2 groups 100 projects and 100 merge requests in the last 90 days in each project = 1 + 1 + 6 + 500 + 10000 + 10000 + 10000 + 2 + 100 + 100 = 30710.