GitLab is an all-in-one DevOps solution with a growing feature set. But as more features are added to the application, more memory is required. Some users have reportedly elected to migrate to other tools because the memory footprint required to run a minimum GitLab instance was exorbitant:
“GitLab is great and I have used it for years but I recently switched to Gogs for self-hosted repositories because it is much faster, easier to set up, and walk in a park to maintain. It doesn't have all the features (bloat) that GitLab has but it can probably satisfy >95% of Git users.” – Jnr on HackerNews
“If GitLab grows any more features I'll be moving away simply to ensure confidence that I understand my own infrastructure in the limited time I have to maintain it. It's the weirdest kind of success problem to have, but the truth is if it wasn't such a pain to make the move, I'd have transitioned away from GitLab six months ago.” – Sir_Substance on HackerNews
Step 1: Establish priorities to solve our memory problem
We created the GitLab Memory team to tackle this performance challenge. The aim of the Memory team is to reduce the minimum instance for GitLab from 8GB to 1GB of RAM. By reducing the memory required to run GitLab to 1GB, our application can run anywhere, even on inexpensive commodity computers like an unaltered Raspberry Pi 3 Model B+.
There is no quick fix for reducing GitLab’s memory footprint, but the team has started by investigating memory and performance bottlenecks, gathering data, and prioritizing activities for the next three to four months based on these results.
“We know we have memory issues to address, but we need more data to determine the source, the impact and how to best approach the problem,” says Craig Gomes, memory engineering manager.
Kamil Trzciński, distinguished engineer and memory specialist at GitLab, says the top three priorities for the Memory team fall into three distinct buckets:
- Moving over to Puma
- Perform the low-level exercise by optimizing endpoints
- Improving our development practices
Migrating from Unicorn to Puma
Preliminary research shows that the bulk of GitLab’s memory usage comes from running web application processes on Unicorn.
“Each Web application process (Unicorn) can take 500 MB of RAM, and it can handle a single request at a time. The more users and traffic we need to support, the more processes and hence RAM we need,” says Stan Hu, engineering fellow at GitLab.
One of the first projects the Memory team is tackling is testing to see if migrating from Unicorn to Puma will reduce GitLab’s memory footprint. Both Unicorn and Puma are multi-threaded HTTP web servers that run on Rails, but unlike Unicorn, Puma is threaded and does not require as much memory.
The Memory team has successfully configured Puma to work on dev.gitlab.com to test its functionality and measure its memory reduction. The next big project in this domain is to enable Puma on GitLab.com.
Dig deeper into what's causing memory issues for GitLab.com
Before GitLab is able to run on less memory, the team needs to fix the memory problems we know about already on GitLab.com. One of these problems is the memory killer on open source background processor, Sidekiq.
"If a Sidekiq job runs, takes too much memory, and then gets killed, jobs in the queue will be retried indefinitely," says Stan. The team is working to fix this, along with other priority one problems with memory usage in project import and exports in the 12.3 release.
Improve development practices around memory usage
The Memory team created a massive epic that aims to capture related development work focusing on making improvements to internal dev practices around code complexity and memory usage.
"The reason behind that is to enable everyone during development to understand the impact of introducing new changes to the application," says Kamil in the epic. Some of the projects they are working on for the 12.3 release include testing more endpoints using typical GitLab user scenarios (e.g. commenting on a MR) and set up a performance monitoring solution across different environments.
Step 2: Create a team to fix the memory problem
We need a specialized engineering team to assess the scope of the problem and identify solutions to reduce GitLab’s memory requirements.
“Right now we have a very small team with two brand new team members,” says Craig. “The team is getting up to speed quickly and there is a lot of excitement about the potential of the team that more work keeps coming our way. It's a great challenge to have, and having more experienced engineers on the team will help us to achieve our goals.”
The current memory team is small but mighty. We have Craig, the engineering manager, and three engineers on the permanent memory team: Kamil, Qingyu Zhao, and Aleksei Lipniagov. The team works closely with senior product manager for distribution and memory, Larissa Lane. We’re looking for more qualified people to join our team.
The Memory team is actively hiring engineers to help us enhance GitLab’s performance, but we have a high rejection rate because we require a specific, hard-to-find skill set. A top priority for the Memory team is hiring at least one senior engineer in FY20-Q3, which will allow us to take on a bigger workload as we move toward our goal of getting GitLab running on less than 1GB.
Follow along with the Memory team by subscribing to their channel on GitLab Unfiltered.
Cover photo by Arie Wubben on Unsplash
