Long texts¶
Why a long field used to come back empty¶
Every translation provider caps the size of a single request. DeepL counts bytes, Google Cloud counts code points, an LLM counts tokens — and the free Google endpoint refuses much past a few thousand characters. Send a 20 000-character article as one string and you get a truncated answer, an error, or nothing at all.
Worse, the failure is silent: an empty column looks exactly like a column nobody translated yet.
What happens now¶
A field of any length is cut into pieces that fit the backend's budget, on the most natural boundary available:
- paragraph breaks (
\n\n), - then sentence ends (
.,!,?,…, and their CJK counterparts), - then word boundaries,
- then, as a last resort for a single unbreakable token, on characters.
The pieces are translated, then glued back. The reassembly is lossless: the whitespace, the line breaks and the paragraph structure of the source are reproduced exactly.
from django_traduire.chunking import chunk_text
chunks = chunk_text(article_body, 3000)
assert "".join(chunks) == article_body # always true
Nothing to enable: it is on for every backend, including custom ones.
Choosing the budget¶
Each backend declares what one request accepts:
| Backend | max_chars |
max_texts |
|---|---|---|
| Google (free) | 3 000 | 1 |
| DeepL | 30 000 | 50 |
| Google Cloud | 28 000 | 100 |
| OpenAI | 5 000 | 20 |
Override it globally when a provider throttles you, or when you are on a slow link:
Or per backend instance:
TRADUIRE = {
"BACKEND": "django_traduire.backends.google_free.GoogleFreeBackend",
"BACKEND_OPTIONS": {
"max_chars": 1500,
"rate_limit": 0.5, # seconds between two requests
},
}
LLM backends
Keep max_chars low for OpenAI-compatible backends. The answer is at least
as long as the input, and a request that runs past the model's output limit
comes back truncated mid-sentence.
Columns that are too small¶
A translation is often longer than its source — German is famously so. Writing
a 240-character translation into a CharField(max_length=200) raises a
database error on PostgreSQL, and silently truncates on MySQL.
django-traduire measures the column before writing:
TRADUIRE = {
"ON_TOO_LONG": "truncate", # cut on a word boundary and log a warning (default)
# "ON_TOO_LONG": "skip", # leave the column empty and log a warning
# "ON_TOO_LONG": "error", # raise TraduireError
}
Whatever the strategy, the event is logged on the django_traduire logger with
the model, the column, and the two sizes.
Bounding a run¶
A translation run costs money and time per row. Both the Python API and the management command take a hard bound: