Skip to content

[scenario tests stability] Pipeline ClusterTopologyRefreshIT verification to fit the test timeout - #4713

Merged
uglide merged 2 commits into
redis:masterfrom
ggivo:topic/ggivo/scenario-stability-ctr-pipelined-verify
Aug 28, 2026
Merged

uglide merged 2 commits into
redis:masterfrom
ggivo:topic/ggivo/scenario-stability-ctr-pipelined-verify

Conversation

@ggivo

@ggivo ggivo commented Aug 28, 2026 •

Copy link
Copy Markdown
Collaborator

Pipeline ClusterTopologyRefreshIT verification to fit the test timeout

What

Replaces ClusterTopologyRefreshIT.testWithPool's sequential per-key exists() verification loop with pipelined batches (500 keys per sync()), keeping an exact per-key failure message.

Why

The test writes one key per round trip while the fault-injection reshard+failover sequence runs, then verified every key with a sequential exists() — another full round trip per key. Both phases scale linearly with the action's duration, so the total test time is roughly twice the action duration and exceeds the 300 s global test timeout (junit-platform.properties) whenever the action takes ~150 s or more.

The nightly scenario runs fail consistently on this:

ClusterTopologyRefreshIT.testWithPool » Timeout testWithPool() timed out after 300 seconds

Measured on the scheduled runs: a passing run spent 120 s in the action and 116 s verifying 2180 keys (237 s total, 79% of the budget); a failing run spent 174 s in the action, leaving the ~156 s verification no room.

With pipelined verification the phase collapses from n × RTT (~2 minutes at ~3000 keys) to a few round trips, so the test time tracks the action duration alone.

Notes

  • The multi-key EXISTS key [key ...] form is not an option in cluster mode (cross-slot); the per-key Response<Boolean> also keeps the exact missing key in the failure message.
  • Test-only change; no client behavior affected.

Note

Low Risk
Test-only change to verification timing; no production client behavior is modified.

Overview
ClusterTopologyRefreshIT.testWithPool no longer verifies written keys with a sequential exists() per key. It batches exists() through a pipeline, sync()ing every 500 keys, and still asserts each key with a missing key {i} message on failure.

This shortens the post–fault-injection verification phase so nightly runs stay under the 300 s JUnit timeout when resharding/failover runs long and thousands of keys were written.

Reviewed by Cursor Bugbot for commit 511017c. Bugbot is set up for automated code reviews on this repo. Configure here.

The test writes one key per round trip while the fault-injection
reshard+failover runs, then verified each key with a sequential
exists() — another full round trip per key. Both phases scale with the
action's duration and together exceed the 300s global test timeout
whenever the action takes ~150s or more. Pipelined batches collapse
verification to a few round trips, so the test time tracks the action
duration alone.
@ggivo ggivo added the testing label Aug 28, 2026
@ggivo ggivo changed the title Pipeline ClusterTopologyRefreshIT verification to fit the test timeout [scenario tests stability] Pipeline ClusterTopologyRefreshIT verification to fit the test timeout Aug 28, 2026
@ggivo

ggivo commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

run scenario tests

@uglide

uglide commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Jedis Scenario Tests (RE 8.0.16)

Tests Passed ✅ Failed ❌ Skipped ⏭️ Flaky 🍂 Duration ⏱️
12 12 0 0 0 not captured

---- Details for maintainers

Github Test Reporter by CTRF 💚

@uglide

uglide commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Jedis Scenario Tests (RE 100.0.20)

Tests Passed ✅ Failed ❌ Skipped ⏭️ Flaky 🍂 Duration ⏱️
12 12 0 0 0 not captured

---- Details for maintainers

Github Test Reporter by CTRF 💚

@uglide

uglide commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Jedis Scenario Tests (RE 8.0.22)

Tests Passed ✅ Failed ❌ Skipped ⏭️ Flaky 🍂 Duration ⏱️
12 11 1 0 0 not captured

---- Details for maintainers

Github Test Reporter by CTRF 💚

@ggivo
ggivo requested a review from uglide August 28, 2026 08:02
@uglide
uglide merged commit bdbff2d into redis:master Aug 28, 2026
19 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants