DEV Community

Cover image for Cross-slot by default: moving Redis Cluster permission into ClusterUtil.batch
Priyanshu Jha
Priyanshu Jha

Posted on

Cross-slot by default: moving Redis Cluster permission into ClusterUtil.batch

Redis Cluster does not let you run a multi-key command unless every key hashes to the same slot. That's a good default. It's also easy to forget, which is why GitLab has a validator that raises if you pipeline keys across slots, and a helper to allow it when you mean to.

This is the story of gitlab-org/gitlab!265488. It moves that permission into Gitlab::Redis::ClusterUtil.batch, so the methods that already batch across slots don't make every caller opt in. I'll cover the issue, the change I shipped, the wrappers I deliberately left alone, and the review that kept the parent issue open.

The issue: a permission every caller had to remember

The issue was gitlab-org/gitlab#474372, opened by Sylvester Chin. It started as a suggestion on an earlier MR, !159542: if ClusterUtil is the place that pipelines keys, ClusterUtil should be the place that allows cross-slot commands.

Gitlab::Redis::ClusterUtil.batch is used by batch_get, batch_del, and batch_unlink. Those methods take a list of keys, slice them, and pipeline a command for each slice. The keys can hash to different slots. That's expected. What's less expected is that every caller still had to wrap the call:

Gitlab::Instrumentation::RedisClusterValidator.allow_cross_slot_commands do
  Gitlab::Redis::ClusterUtil.batch_get(keys, redis)
end
Enter fullscreen mode Exit fullscreen mode

Forget the wrapper, and you get a CrossSlotError for a method whose whole job is to batch keys. The issue also listed a second, broader cleanup: Rails.cache.delete_multi, fetch_multi, and read_multi call sites could drop their wrappers once the permission lived lower down.

Callers wrapping ClusterUtil, and the wrapper moving inside batch

The plan, and what I decided not to do in the same MR

I commented on the issue with this plan:

Plan: wrap the pipelined calls in Gitlab::Redis::ClusterUtil.batch (used by batch_get/batch_del/batch_unlink) with Gitlab::Instrumentation::RedisClusterValidator.allow_cross_slot_commands, so callers no longer need to wrap them. I'll then drop the now-redundant wrappers at call sites that only call ClusterUtil, and add a spec showing the batch methods work without an outer allow_cross_slot_commands block. I'll keep the diff small and leave the Rails.cache.*_multi callers for a follow-up unless you'd like them in the same MR.

The contributor platform assigned me, and I opened the MR with a smaller scope than that plan's middle sentence. I did not drop the wrappers at existing call sites.

The reason is in the MR description. Callers such as Gitlab::AvatarCache, Gitlab::SetCache, Gitlab::RepositoryHashCache, Gitlab::ReactiveCacheSetCache, and Gitlab::SidekiqStatus don't only go through ClusterUtil. Their non-cluster branches still run commands like redis.unlink(*keys) and redis.mget(*keys), which need the wrapper on their own. Removing it because the cluster path no longer needs it would have been a nice-looking cleanup and a real bug on the other branch.

The Rails.cache.*_multi callers mentioned in the issue were left for a follow-up, as planned.

The change

The production change is a wrap around the existing loop in ClusterUtil.batch:

def batch(entries, redis)
  Gitlab::Instrumentation::RedisClusterValidator.allow_cross_slot_commands do
    entries.each_slice(pipeline_batch_size).flat_map do |subset|
      redis.pipelined do |pipeline|
        yield pipeline, subset
      end
    end
  end
end
Enter fullscreen mode Exit fullscreen mode

batch_get, batch_del, and batch_unlink all go through this method, so they inherit the permission. The validator tracks nesting, so a caller that still has its own allow_cross_slot_commands block keeps working. I used that fact rather than touching those callers.

The specs had to change in two ways. First, the existing batch examples no longer wrap themselves in allow_cross_slot_commands. If the production change were missing, those examples would fail, which is the point:

it 'gets multiple keys without an outer allow_cross_slot_commands block' do
  Gitlab::Redis::Cache.with do |c|
    expect(described_class.batch_get(array, c)).to eq(array)
  end
end
Enter fullscreen mode Exit fullscreen mode

Second, I added examples on .batch itself that check the flag is on while the block runs, and that it is restored afterwards, including when the block raises:

it 'allows cross slot commands while yielding and restores the state afterwards' do
  allowed = []

  Gitlab::Redis::Cache.with do |c|
    described_class.batch(%w[a b c], c) do |_pipeline, _subset|
      allowed << validator.allow_cross_slot_commands?
      []
    end
  end

  expect(allowed).to eq([true])
  expect(validator.allow_cross_slot_commands?).to be(false)
end

it 'restores the state when the block raises' do
  Gitlab::Redis::Cache.with do |c|
    expect { described_class.batch(%w[a], c) { raise 'boom' } }.to raise_error('boom')
  end

  expect(validator.allow_cross_slot_commands?).to be(false)
end
Enter fullscreen mode Exit fullscreen mode

A helper that flips a process-wide (or thread-local) flag and does not restore it on an exception is a gift to the next test, and to the next request.

The description also has a local validation step: in a Rails console, call batch_get(%w[a b c], redis) inside Gitlab::Redis::Cache.with without wrapping it, and it should no longer raise CrossSlotError.

The review: LGTM, and don't close the issue

I marked the MR ready, then sent a gentle ping a few days later. Saahil Sabu Hameed reviewed it, approved it, and asked for one description change:

The issue also asks for a cleanup.

All usages of Rails.cache.delete_multi or .fetch_multi or .read_multi can be cleaned up to not need Gitlab::Instrumentation::RedisClusterValidator.allow_cross_slot_commands.

Lets update the MR description to say "Related to" instead of "Closes"? You can open up another MR for the cleanup if you are interested in working on that as well.

That's the right call. This MR made the batch helpers safe to call on their own. It did not finish the issue. Closing #474372 would have made the leftover work harder to find. I updated the description to "Related to," and said I was happy to take the Rails.cache cleanup as a follow-up.

Saahil asked Drew Blessing for the backend maintainer review. Drew approved it and added it to the merge train. The MR merged on 3 November 2026.

The leftover wrappers, and the issue left open for the Rails.cache cleanup

What I took away

1. Put the permission next to the operation that needs it. If ClusterUtil.batch is the thing that pipelines keys across slots, callers shouldn't have to remember to allow that. The next call site will forget.

2. A wrapper that still protects another branch is not redundant. I had planned to drop wrappers at call sites that "only" use ClusterUtil. Looking at those files, they also have non-cluster mget and unlink paths. I left them alone. A cleanup that is only safe on one branch is not a cleanup.

3. Don't close an issue you only partly solved. "Related to" is more honest than "Closes" when the issue also asks for a Rails.cache cleanup. Saahil caught that in review.

4. If you flip a flag, test that it flips back when the block raises. The happy-path restore is easy to write and easy to get right. The exception path is the one that leaks.

5. Nesting matters. Because the validator already tracked nested allow_cross_slot_commands blocks, I could add the inner wrap without breaking callers that still have an outer one. That's what made a small MR safe.

Links

Thanks to Sylvester Chin for the issue, Saahil Sabu Hameed for the review and for keeping the leftover work visible, and Drew Blessing for the maintainer review.

If you've been thinking about contributing to GitLab, the community issues are a great place to start: contributors.gitlab.com.

Top comments (0)