You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[feat] Omit padded tokens in generate() of huggingface transformers #17
Currently generate() of huggingface transformers pads all requests in a batch to the same length both in prefill and in decode.
This leads to offloading meaningless padded tokens, which can be a lot if some requests are notably longer than others.
Currently generate() of huggingface transformers pads all requests in a batch to the same length both in prefill and in decode.
This leads to offloading meaningless padded tokens, which can be a lot if some requests are notably longer than others.