We’ve set up several initiatives aimed at testing and improving the performance of GitLab, which is why the Quality team built a new tool to test GitLab's performance.
Performance testing is an involved process and distinct from other testing disciplines. The strategies and tooling in this space are specialized and require dedicated resources to achieve results. When I joined the company and became the first member of this team, the task was to expand our nascent performance efforts to a much larger scale. For this, we needed to build out a new tool that we aptly named the GitLab Performance tool (GPT).
We're happy to announce the general release of GPT. In this blog post, we'll share how GPT is used to performance test GitLab, and how you can use it as well to test your own environments.
However, before we get into what the GPT is, we need to first touch on what we use it with.
Reference Architectures and test data
In our experience, the challenging part of performance testing isn’t actually to do with the testing itself, but instead configuring the right environments and data to test against.
As such, one of the initiatives we’ve been driving is the design of several GitLab Reference Architectures that can handle large numbers of users. We wanted to create these architectures as a way to standardize our recommended configurations to ensure we were presenting customers with options for performant, scalable, and highly available GitLab setups.
In order to create a tool like this, we needed to add realistic data into these environments to test against, e.g., large projects with commits and merge requests. As a first iteration, we started with our very own GitLab project.
Once we got our environments running and configured we were ready to test them with the GPT.
What is the GitLab Performance tool (GPT)?
The GPT can be used to run numerous load tests to verify the performance of any GitLab environment. All that’s required is to a knowledge of what throughput the intended environment can handle (as requests per second) and to ensure that the environment has the necessary data prepared.
The GPT is built upon one of the leading tools in the industry, k6. Here are some examples of what the GPT provides:
- A broad test suite that comes out-of-the-box and covers various endpoints across the GitLab product with added ability to add your own custom tests as desired. See the latest out-of-the-box test details with more being added frequently.
- Options for customizing test runs, such as specifying desired GitLab environment data or defining what throughput to use with default examples given.
- Ability to run multiple tests sequentially as well as be selective about which are chosen.
- Enhanced reporting and logging.
- Built-in test success thresholds based on time to first byte, throughput achieved and successful responses.
The talented team at Load Impact created k6, which is the core of the GPT. We realized quickly that we didn’t need to reinvent the wheel because k6 met most of our needs: It is written in Go, so is very performant and is an open source solution. Thanks to the team for not only developing k6 but also for reaching out to us soon after we started to collaborate.
How we use GPT
We use the GPT in several automated GitLab CI pipelines for quick feedback on how GitLab is performing. The CI pipelines typically run daily or weekly against our reference architecture environments, which themselves are running on the latest pre-release code. We review the test results as they come in and then investigate any failures. In line with our Transparency value, we also publish all of the latest results for anyone to view on the GPT wiki.
The GPT is also used in a comparison test pipeline to see how GitLab’s performance changes in every release cycle. These results are important because they show the whole picture of our performance evolution. The benchmark comparison results are also available on the GPT wiki.
By using the GPT, we’ve been able to identify several performance pain points of GitLab and collaborate with our dev teams to prioritize improvements. The process has been fruitful so far and we’re excited to already see improvements in the performance numbers with each release of GitLab. The 12.6 release for example showed several notable improvements across the board, one even as high as a 92% reduction! You can see the issues we've raised so far through this work over on our issue tracker.
How you can use GPT
We decided early that we wanted to follow the same open source principles as our main product, so we build the GPT with all users in mind rather than making it a strictly internal tool. So not only do we let others use it, we encourage it! This is beneficial for us and customers, as we receive feedback from diverse viewpoints that we hadn’t considered. Some examples of this are improving the recommended spec guidelines based on throughput or making it easier for users who have private clouds to use the GPT offline.
If you want to use the GPT for yourself, the best place to start is with its documentation. As mentioned earlier, most of the effort to use the GPT is preparing the intended environment. The docs will take you through this along with how to use the tool itself.
The GPT in action
Finally after writing all about the GPT we should actually show it in action. Here's how it looks when running a load test for the List Group Projects API against our 10k Reference Architecture:
Read the GPT documentation for more details on output and results.
What’s next?
Our aim is to make GitLab’s performance best in class. This is only the start of our performance testing journey with GPT and we are excited about the additional ways we can continue to help improve the customer experience.
Some examples of our plans for the next few releases include expanding test coverage to more of GitLab’s features and entry points (API, Web, Git) and expanding our work on the reference architectures, test data, and user behavior patterns to be as representative and realistic as possible.
Share your feedback and/or suggestions on GPT here or on our GPT project! We welcome your ideas or contributions.
Cover image by Taylor Vick on Unsplash.
