POC: Optimize memory usage of CI variables collection

Information

This is a POC. The work here will be broken down into multiple MRs with a feature flag to avoid incidents.

MRs that are created from this POC:

1. Rename variable-models to_runner_variable to to... (!172405 - merged)
2. Optimize memory usage of CI variables collectio... (!173126 - merged)

What does this MR do and why?

• Remove redundant usage of to_runner_variable/to_runner_variables because it tries to modify the hash.
• Collection#to_hash: We avoid creating an intermediate hash and then converting it to a HashWithIndifferentAccess.
• Collection::Item#to_runner_variable: We do not create another hash anymore.
• Collection::Item#fabricate: We do not duplicate the object anymore.

Feature flag: ci_optimize_memory_for_variables #502800 (closed)

References

Related to #499707 (closed)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

🚀 Total memory reduction for this example pipeline; ~200 MB.

# RAILS_PROFILE=true GITALY_DISABLE_REQUEST_LIMITS=true rails console

require 'memory_profiler'

ActiveRecord::Base.logger = nil
project = Project.find_by_full_path('root/gitlab-mirror')
user = project.first_owner
merge_request = project.merge_requests.find_by_iid(1)

# Warmup
Ci::CreatePipelineService
 .new(project, user, ref: merge_request.source_branch)
 .execute(:merge_request_event, merge_request: merge_request); nil

report = MemoryProfiler.report do
 Gitlab::SafeRequestStore.ensure_request_store do
 Ci::CreatePipelineService
 .new(project, user, ref: merge_request.source_branch)
 .execute(:merge_request_event, merge_request: merge_request); nil
 end
end; nil

output = File.open('tmp/memory-profile-report.txt', 'w')
report.pretty_print(output, detailed_report: true, scale_bytes: true, normalize_paths: true)

::Ci::DestroyPipelineService.new(project, user).execute(Ci::Pipeline.last)
::Ci::DestroyPipelineService.new(project, user).execute(Ci::Pipeline.last)

Before

#
# Note: I redacted some parts related to the gems and the Rails framework.
# also, the output is shortened for readability.
#

Total allocated: 1.30 GB (12974240 objects)
Total retained: 29.67 MB (335085 objects)

allocated memory by gem
-----------------------------------
 675.48 MB gitlab/lib

...

allocated memory by file
-----------------------------------
 253.68 MB gitlab/lib/gitlab/ci/variables/collection/item.rb
 143.58 MB gitlab/lib/gitlab/ci/variables/collection.rb
 51.66 MB gitlab/lib/gitlab/config/entry/configurable.rb
 20.89 MB gitlab/lib/gitlab/ci/pipeline/expression/lexeme/base.rb

...

allocated memory by location
-----------------------------------
 107.12 MB gitlab/lib/gitlab/ci/variables/collection/item.rb:64
 70.22 MB gitlab/lib/gitlab/ci/variables/collection.rb:28
 57.66 MB gitlab/lib/gitlab/ci/variables/collection.rb:82
 45.70 MB gitlab/lib/gitlab/config/entry/configurable.rb:67
 42.35 MB gitlab/lib/gitlab/ci/variables/collection/item.rb:17
 42.35 MB gitlab/lib/gitlab/ci/variables/collection/item.rb:80
 41.32 MB gitlab/lib/gitlab/ci/variables/collection/item.rb:76
 20.10 MB gitlab/lib/gitlab/ci/variables/collection/item.rb:72

...

After

#
# Note: I redacted some parts related to the gems and the Rails framework.
# also, the output is shortened for readability.
#

Total allocated: 1.08 GB (11171148 objects)
Total retained: 29.67 MB (335082 objects)

allocated memory by gem
-----------------------------------
 495.88 MB gitlab/lib

...

allocated memory by file
-----------------------------------
 112.44 MB gitlab/lib/gitlab/ci/variables/collection.rb
 105.24 MB gitlab/lib/gitlab/ci/variables/collection/item.rb
 51.66 MB gitlab/lib/gitlab/config/entry/configurable.rb
 20.89 MB gitlab/lib/gitlab/ci/pipeline/expression/lexeme/base.rb

...