Optimize memory usage of CI variables collection/item

What does this MR do and why?

Optimizations:

• Remove redundant usage of to_runner_variable/to_runner_variables because it tries to modify the hash.
• Collection#to_hash: We avoid creating an intermediate hash and then converting it to a HashWithIndifferentAccess.

Also, the usage of context.variables.sort_and_expand_all is collected into a single method which will enable us to make more optimizations.

Feature flag: ci_optimize_memory_for_variables #502800 (closed).

References

Related to #499707 (closed)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

🚀 Total memory reduction for this example pipeline; ~140 MB.

# RAILS_PROFILE=true GITALY_DISABLE_REQUEST_LIMITS=true rails console

require 'memory_profiler'

ActiveRecord::Base.logger = nil
project = Project.find_by_full_path('root/gitlab-mirror')
user = project.first_owner
merge_request = project.merge_requests.find_by_iid(1)

# Warmup
Ci::CreatePipelineService
 .new(project, user, ref: merge_request.source_branch)
 .execute(:merge_request_event, merge_request: merge_request); nil

report = MemoryProfiler.report do
 Gitlab::SafeRequestStore.ensure_request_store do
 Ci::CreatePipelineService
 .new(project, user, ref: merge_request.source_branch)
 .execute(:merge_request_event, merge_request: merge_request); nil
 end
end; nil

output = File.open('tmp/memory-profile-report.txt', 'w')
report.pretty_print(output, detailed_report: true, scale_bytes: true, normalize_paths: true)

::Ci::DestroyPipelineService.new(project, user).execute(Ci::Pipeline.last)
::Ci::DestroyPipelineService.new(project, user).execute(Ci::Pipeline.last)

Before

#
# Note: I redacted some parts related to the gems and the Rails framework.
# also, the output is shortened for readability.
#

Total allocated: 1.30 GB (13038034 objects)
Total retained: 29.68 MB (335107 objects)

allocated memory by gem
-----------------------------------
 675.90 MB gitlab/lib

...

allocated memory by file
-----------------------------------
 253.68 MB gitlab/lib/gitlab/ci/variables/collection/item.rb
 143.58 MB gitlab/lib/gitlab/ci/variables/collection.rb
 51.66 MB gitlab/lib/gitlab/config/entry/configurable.rb
 20.89 MB gitlab/lib/gitlab/ci/pipeline/expression/lexeme/base.rb

...

allocated memory by location
-----------------------------------
 107.12 MB gitlab/lib/gitlab/ci/variables/collection/item.rb:67
 70.22 MB gitlab/lib/gitlab/ci/variables/collection.rb:28
 57.66 MB gitlab/lib/gitlab/ci/variables/collection.rb:88
 45.70 MB gitlab/lib/gitlab/config/entry/configurable.rb:67
 42.35 MB gitlab/lib/gitlab/ci/variables/collection/item.rb:17
 42.35 MB gitlab/lib/gitlab/ci/variables/collection/item.rb:87
 41.32 MB gitlab/lib/gitlab/ci/variables/collection/item.rb:83
 20.10 MB gitlab/lib/gitlab/ci/variables/collection/item.rb:79

...

After

#
# Note: I redacted some parts related to the gems and the Rails framework.
# also, the output is shortened for readability.
#

Total allocated: 1.13 GB (11763652 objects)
Total retained: 29.68 MB (335105 objects)

allocated memory by gem
-----------------------------------
 538.00 MB gitlab/lib

...

allocated memory by file
-----------------------------------
 146.68 MB gitlab/lib/gitlab/ci/variables/collection/item.rb
 112.44 MB gitlab/lib/gitlab/ci/variables/collection.rb
 51.66 MB gitlab/lib/gitlab/config/entry/configurable.rb
 20.89 MB gitlab/lib/gitlab/ci/pipeline/expression/lexeme/base.rb

...