Skip to content

get_repetition_penalty_reward accepts non-positive ngram_size #7015

Closed as not planned

Description

@qsxustc

Reproduction

ngram_size values of zero or less are accepted when constructing the reward function, but fail later during reward computation.

from trl.rewards import get_repetition_penalty_reward

reward_fn = get_repetition_penalty_reward(ngram_size=0)
reward_fn([[1, 2, 3]])

outputs:

Traceback (most recent call last):
  File "reproduce.py", line 4, in <module>
    reward_fn([[1, 2, 3]])
  File "trl/rewards/other_rewards.py", line 78, in __call__
    scaling = 1 - len(set(ngrams)) / len(ngrams)
                  ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~
ZeroDivisionError: division by zero

Negative values have the same problem because no n-grams are constructed.

Expected behavior: validate ngram_size when constructing the reward function and raise an actionable ValueError unless it is greater than zero. This is consistent with the existing construction-time validation for max_penalty.

System Info

  • Platform: macOS-26.6.2-arm64-arm-64bit-Mach-O
  • Python version: 3.14.6
  • TRL version: 1.13.0.dev0+323efb2
  • PyTorch version: 2.13.0
  • accelerator(s): cpu
  • Transformers version: 5.16.1
  • Accelerate version: 1.14.0
  • Accelerate config: not found
  • Datasets version: 5.0.1
  • HF Hub version: 1.29.0
  • bitsandbytes version: not installed
  • DeepSpeed version: not installed
  • Liger-Kernel version: not installed
  • PEFT version: not installed
  • vLLM version: not installed

Checklist

  • I have checked that my issue isn't already filed (see open issues)
  • I have included my system information
  • Any code provided is minimal, complete, and reproducible (more on MREs)
  • Any code provided is properly formatted in code blocks, (no screenshot, more on code blocks)
  • Any traceback provided is complete

Activity

  1. qgallouedec commented on Sep 16, 2026

    @qgallouedec
    Member

    what does ngram_size=0 or -1 mean?

  2. pravinmountain commented on Sep 18, 2026

    @pravinmountain

    ngram_size is the sliding-window size for building n-grams (zip(ids[0:], ids[1:], ...))

    • ngram_size=0: range(0) is empty, so the list comprehension produces [], and zip() with no arguments yields nothing. ngrams ends up as an empty list, so line 78 computes len(set([])) / len([]) — that's 0 / 0, the ZeroDivisionError in the issue.
    • ngram_size=-1: same thing, because range(-1) is also empty. The guard on line 74 (len(ids) < self.ngram_size) doesn't save us.
      So there's no sensible interpretation for non-positive values; I'd propose raising a ValueError at construction (mirroring the existing max_penalty check).
  3. qgallouedec commented on Sep 18, 2026

    @qgallouedec
    Member

    That's not my question. My question is what does it means, ie why a user would choose such a value? In what real-world training would you use 0 or negative size?

  4. pravinmountain commented on Oct 1, 2026

    @pravinmountain

    Hi, there is no real world use of ngram_size=0 or negative value. Nobody will choose it on purpose.
    ngram_size is the size of the sliding window over token ids. A window of 0 tokens (or -1 token) dont mean anything. It was probably a typo or someone just testing edge case.

    Example with ngram_size=2:

    • [1, 2, 3, 4] gives windows (1,2) (2,3) (3,4). All are diffrent, so no repetition.
    • [5, 5, 5, 5, 5] gives (5,5) (5,5) (5,5) (5,5). 4 windows but only 1 unique, so very repetitive.

    Reward formula is (1 - unique / total) * max_penalty. So:

    • [1, 2, 3, 4] -> 1 - 3/3 = 0 -> reward 0.0
    • [5, 5, 5, 5, 5] -> 1 - 1/4 = 0.75 -> reward -0.75
    from trl.rewards import get_repetition_penalty_reward
    
    reward_fn = get_repetition_penalty_reward(ngram_size=2, max_penalty=-1.0)
    print(reward_fn([[1, 2, 3, 4], [5, 5, 5, 5, 5]]))
    # [0.0, -0.75]

    Now with ngram_size=0:

    ids = [1, 2, 3]
    [ids[i:] for i in range(0)]   # range(0) is empty -> []
    list(zip(*[]))                # zip() with no argument -> []
    len(set([])) / len([])        # 0 / 0 -> ZeroDivisionError

    So it fails for every input, even [[1, 2, 3]] which is a valid input. The input is not the problem, the size is the problem. Negative value also give empty window, same error.

    So the issue is not about supporting some use case. It is only that the error come late and is confusing (ZeroDivisionError). Better to raise a ValueError in the constructor when ngram_size < 1, same like the max_penalty check. Thanks!

  5. albertvillanova commented on Oct 1, 2026

    @albertvillanova
    Member

    Thanks for digging into it. Agreed that ngram_size=0 or a negative value has no meaning and nobody picks it on purpose.

    That is the reason not to add the check rather than a reason to add it. TRL avoids defensive validation for cases that do not occur, and a typo here already fails immediately and points at the line: the first call raises ZeroDivisionError at scaling = 1 - len(set(ngrams)) / len(ngrams). A guard would only reword an error the user already gets.

    max_penalty is validated for the opposite reason: a positive value never raises. It silently flips the sign, so repetition is rewarded instead of penalised and training quietly optimises toward degenerate text. That is the case worth guarding, and it is the only such guard in trl/rewards/.

    Closing as not planned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions