Rich text API¶

Translating a document¶

`django_traduire.richtext` ¶

HTML-aware translation: translate the words, keep the markup.

A rich-text field is not plain text. Sending <table><tr><td>Le cacao</td>… to a provider as-is either strips the tags, scrambles the rows, or comes back truncated because the table alone is larger than one request. This module walks the document, hands the provider prose that fits, and puts the markup back exactly where it was.

The pipeline has one shape, whatever the provider:

:func:tokenize turns the document into a flat list of tokens.
Tokens are grouped into parts: structural markup on one side, runs of translatable content on the other. A heading, a list item and a table cell are each their own run.
Every run is cut into segments that fit the provider's request budget, on token boundaries — never in the middle of a tag, and in inline_html mode never between a tag and the one that closes it.
A segment is sent either with its inline markup replaced by [[n]] placeholders (providers that read plain text, such as the free Google endpoint) or as real inline HTML (DeepL, Google Cloud, an LLM).
Answers are put back in place; the markup is restored byte for byte.

Attributes a reader sees — alt, title, placeholder, aria-label — are translated too, and only their value is rewritten.

Content of <script>, <style>, <code>, <pre>, <template> and their friends is never touched, and neither is Django or Jinja template syntax ({{ … }}, {% … %}) sitting inside prose.

`Part` `dataclass` ¶

A stretch of the document: structural markup, or a translatable run.

Attributes:

Name	Type	Description
`kind`	`str`	`"raw"` for markup that is re-emitted as-is, `"unit"` for a run of content split into :class:`Segment` objects.
`tokens`	`tuple[Token, ...]`	The tokens of the part, in document order.
`segments`	`tuple[Segment, ...]`	The segments of a `"unit"` part, in document order.

Source code in src/django_traduire/richtext.py

@dataclass(frozen=True)
class Part:
    """A stretch of the document: structural markup, or a translatable run.

    Attributes:
        kind: ``"raw"`` for markup that is re-emitted as-is, ``"unit"`` for a
            run of content split into :class:`Segment` objects.
        tokens: The tokens of the part, in document order.
        segments: The segments of a ``"unit"`` part, in document order.
    """

    kind: str
    tokens: tuple[Token, ...]
    segments: tuple[Segment, ...] = field(default=())

`Segment` `dataclass` ¶

A slice of a translatable run that fits one request.

Attributes:

Name	Type	Description
`ident`	`int`	Identifier of the segment inside the document.
`tokens`	`tuple[Token, ...]`	The tokens this segment covers, in document order.
`lead`	`str`	Whitespace stripped from the front of the request text.
`core`	`str`	What is actually sent to the provider.
`trail`	`str`	Whitespace stripped from the end of the request text.
`markup`	`tuple[Token, ...]`	Inline markup set aside, in placeholder order. Empty when the segment is sent as HTML.
`translatable`	`bool`	False when the segment holds no prose at all.

Source code in src/django_traduire/richtext.py

@dataclass(frozen=True)
class Segment:
    """A slice of a translatable run that fits one request.

    Attributes:
        ident: Identifier of the segment inside the document.
        tokens: The tokens this segment covers, in document order.
        lead: Whitespace stripped from the front of the request text.
        core: What is actually sent to the provider.
        trail: Whitespace stripped from the end of the request text.
        markup: Inline markup set aside, in placeholder order. Empty when the
            segment is sent as HTML.
        translatable: False when the segment holds no prose at all.
    """

    ident: int
    tokens: tuple[Token, ...]
    lead: str = ""
    core: str = ""
    trail: str = ""
    markup: tuple[Token, ...] = ()
    translatable: bool = False

`AttributeRequest` `dataclass` ¶

One attribute value waiting for its translation.

Source code in src/django_traduire/richtext.py

@dataclass(frozen=True)
class AttributeRequest:
    """One attribute value waiting for its translation."""

    token: int  #: index of the tag the attribute belongs to
    name: str  #: attribute name, lowercased
    text: str  #: the value to translate

`translate_html(html, translate, *, inline_html=False, max_chars=None, translate_attributes=True)` ¶

Translate the prose of an HTML document, keeping its structure intact.

Parameters:

Name	Type	Description	Default
`html`	`str`	The document to translate.	required
`translate`	`Callable[[list[str]], list[str]]`	Callable receiving a list of strings and returning their translations, in the same order and the same number.	required
`inline_html`	`bool`	True when the provider reads markup. Inline tags are then sent as HTML instead of `[[n]]` placeholders, and their attributes are left to the provider.	`False`
`max_chars`	`int \| None`	Characters one request may carry. Runs longer than this are cut on token boundaries. None means no budget.	`None`
`translate_attributes`	`bool`	Translate `alt`, `title`, `placeholder` and `aria-label` values as well.	`True`

Returns:

Name	Type	Description
`str`	`str`	The document with translated prose and untouched markup.

Raises:

Type	Description
`TypeError`	If `translate` is not callable.
`ValueError`	If `max_chars` is not a positive integer.
`BackendError`	If `translate` returns the wrong number of results.

Source code in src/django_traduire/richtext.py

def translate_html(
    html: str,
    translate: Callable[[list[str]], list[str]],
    *,
    inline_html: bool = False,
    max_chars: int | None = None,
    translate_attributes: bool = True,
) -> str:
    """Translate the prose of an HTML document, keeping its structure intact.

    Args:
        html: The document to translate.
        translate: Callable receiving a list of strings and returning their
            translations, in the same order and the same number.
        inline_html: True when the provider reads markup. Inline tags are then
            sent as HTML instead of ``[[n]]`` placeholders, and their
            attributes are left to the provider.
        max_chars: Characters one request may carry. Runs longer than this are
            cut on token boundaries. None means no budget.
        translate_attributes: Translate ``alt``, ``title``, ``placeholder`` and
            ``aria-label`` values as well.

    Returns:
        str: The document with translated prose and untouched markup.

    Raises:
        TypeError: If ``translate`` is not callable.
        ValueError: If ``max_chars`` is not a positive integer.
        BackendError: If ``translate`` returns the wrong number of results.
    """
    if not callable(translate):
        raise TypeError("translate_html() expects a callable translate()")
    if max_chars is not None and (not isinstance(max_chars, int) or max_chars <= 0):
        raise ValueError(f"max_chars must be a positive integer, got {max_chars!r}")

    parts = build_parts(html, max_chars, inline_html)
    segments = [segment for part in parts for segment in part.segments if segment.translatable]
    requests = _attribute_requests(parts, inline_html) if translate_attributes else []
    texts = [segment.core for segment in segments] + [request.text for request in requests]
    if not texts:
        return _assemble(parts, {}, {})

    answers = _send(translate, texts, max_chars)
    attributes = {
        (request.token, request.name): answer
        for request, answer in zip(requests, answers[len(segments) :], strict=True)
        if answer
    }
    rendered, failed = _render_segments(segments, answers[: len(segments)], inline_html, attributes)
    rendered.update(_fallback(failed, translate, max_chars, attributes))
    return _assemble(parts, rendered, attributes)

`build_parts(html, max_chars=None, inline_html=False)` ¶

Split a document into structural markup and budgeted translatable runs.

Parameters:

Name	Type	Description	Default
`html`	`str`	The document to plan.	required
`max_chars`	`int \| None`	Characters one request may carry, or None for no limit.	`None`
`inline_html`	`bool`	True when the provider reads markup, so inline tags stay in the text instead of becoming placeholders.	`False`

Returns:

Type	Description
`list[Part]`	list[Part]: The document, in order.

Source code in src/django_traduire/richtext.py

def build_parts(html: str, max_chars: int | None = None, inline_html: bool = False) -> list[Part]:
    """Split a document into structural markup and budgeted translatable runs.

    Args:
        html: The document to plan.
        max_chars: Characters one request may carry, or None for no limit.
        inline_html: True when the provider reads markup, so inline tags stay
            in the text instead of becoming placeholders.

    Returns:
        list[Part]: The document, in order.
    """
    tokens = tokenize(html)
    idents = count()
    parts: list[Part] = []
    buffer: list[Token] = []
    for token, skipped in zip(tokens, _skip_flags(tokens), strict=True):
        if not _is_boundary(token, skipped):
            buffer.append(token)
            continue
        if buffer:
            parts.append(_unit_part(buffer, max_chars, inline_html, idents))
            buffer = []
        parts.append(Part("raw", (token,)))
    if buffer:
        parts.append(_unit_part(buffer, max_chars, inline_html, idents))
    return parts

`split_html(html, max_chars)` ¶

Cut html into fragments of at most max_chars, on tag boundaries.

The concatenation of the fragments is exactly html. Cuts are taken between top-level elements first; a single element larger than max_chars — a long table, a nested list — is opened up and cut one level deeper, as many times as it takes. Fragments produced that way are slices of the document rather than balanced elements, which is why :func:translate_html plans its own segments instead of calling this.

Raises:

Type	Description
`TypeError`	If `html` is not a string.
`ValueError`	If `max_chars` is not a positive integer.

Source code in src/django_traduire/richtext.py

def split_html(html: str, max_chars: int) -> list[str]:
    """Cut ``html`` into fragments of at most ``max_chars``, on tag boundaries.

    The concatenation of the fragments is exactly ``html``. Cuts are taken
    between top-level elements first; a single element larger than
    ``max_chars`` — a long table, a nested list — is opened up and cut one
    level deeper, as many times as it takes. Fragments produced that way are
    slices of the document rather than balanced elements, which is why
    :func:`translate_html` plans its own segments instead of calling this.

    Raises:
        TypeError: If ``html`` is not a string.
        ValueError: If ``max_chars`` is not a positive integer.
    """
    if not isinstance(html, str):
        raise TypeError(f"split_html() expects a string, got {type(html).__name__}")
    if not isinstance(max_chars, int) or max_chars <= 0:
        raise ValueError(f"max_chars must be a positive integer, got {max_chars!r}")
    if len(html) <= max_chars:
        return [html]

    cuts = _cut_points(html)
    units = [html]
    for level in range(MAX_SPLIT_DEPTH + 1):
        offsets = sorted(
            {0, len(html), *(cut for cut, depth in cuts if depth <= level and 0 < cut < len(html))}
        )
        units = [html[start:end] for start, end in pairwise(offsets)]
        if max(len(unit) for unit in units) <= max_chars:
            break
    return _pack_fragments(units, max_chars)

`tokenize(html)` ¶

Parse html into tokens, refusing a document that is not editorial.

Raises:

Type	Description
`TypeError`	If `html` is not a string.
`TraduireError`	If the document holds more than :data:`MAX_TOKENS_PER_DOCUMENT` tokens.

Source code in src/django_traduire/richtext.py

def tokenize(html: str) -> list[Token]:
    """Parse ``html`` into tokens, refusing a document that is not editorial.

    Raises:
        TypeError: If ``html`` is not a string.
        TraduireError: If the document holds more than
            :data:`MAX_TOKENS_PER_DOCUMENT` tokens.
    """
    tokens = _tokenize(html)
    if len(tokens) > MAX_TOKENS_PER_DOCUMENT:
        raise TraduireError(
            f"Document holds {len(tokens)} tokens, over the "
            f"{MAX_TOKENS_PER_DOCUMENT} limit; translate it in smaller fields."
        )
    return tokens

Reading and writing markup¶

`django_traduire.htmltokens` ¶

Read a rich-text document as a flat list of tokens, and write it back.

Everything that knows what an HTML tag is lives here: which tags open a block, which ones hide code rather than prose, which attributes carry words a reader sees. :mod:django_traduire.richtext builds on it and never parses markup itself.

The contract of this module is that a document survives a round trip::

"".join(render_token(token) for token in tokenize(html)) == html

up to entity decoding — ô comes back as ô,   comes back as  . Tags, attributes, comments, doctypes and processing instructions are re-emitted exactly as they were written.