Large language models can translate a sentence fluently, but the translation may still be culturally wrong.

For example, an AI system may:

• Translate a kinship term incorrectly.
• Remove the social meaning of an honorific.
• Replace a local festival or food with a foreign equivalent.
• Translate an idiom word-for-word and lose its real meaning.

These translations look correct to the reader because they are grammatically accurate. However, they may misrepresent the culture or original meaning. The research calls this the “fluent-but-culturally-inappropriate” translation failure.

Most common translation metrics, such as BLEU, COMET, chrF, and BERTScore, mainly measure language quality and similarity. They may not identify whether a translation correctly represents the cultural meaning of the original sentence.

Previous studies have found that AI models sometimes contain useful information inside their hidden layers, even when this information does not appear correctly in the final answer.

For example, hidden layers have been used to detect:

• Whether a statement is true or false.
• Whether an answer is likely to be correct.
• Which culture or cultural tradition is represented in the input.
• Whether the model has recognized a cultural reference.

One cited study found that cultural information could be identified from a model’s internal representations with approximately 79% peak accuracy, while the model’s free-generated answers achieved only about 26% accuracy. This suggests that the model may internally recognize cultural information but fail to express it correctly in its final output.

The paper also found that important information is not always strongest in the final layer of a model. In some cases, the middle layers contain a clearer signal than the final layer. This is why the authors propose analyzing every layer instead of examining only the model’s final output.

However, the authors clarify that they did not conduct a new translation experiment in this paper. They reanalyzed results from earlier studies and used them to design a future experiment.

The paper proposes a new dataset and evaluation system called the Serideo Cultural Translation Evaluation Set.

The dataset would focus on:

• Hindi
• Bengali
• Tamil
• Telugu
• Hinglish as an additional robustness condition

The proposed research process is:

1. Create English sentences containing cultural references such as idioms, festivals, food, kinship terms, honorifics, and local customs.
2. Translate these sentences using models such as Qwen and Llama.
3. Check whether each translation is linguistically fluent.
4. Keep only the fluent translations for the main analysis.
5. Ask native speakers whether the translations are culturally correct.
6. Extract hidden-layer representations from the AI models.
7. Train small classifiers, called probes, to identify culturally correct and culturally incorrect translations.
8. Test whether the probe works across different Indian languages.
9. Test the system with paraphrased sentences, different prompts, another model, and Hinglish.

The most important design decision is to compare translations that are all fluent. This ensures that the probe detects cultural correctness, rather than simply detecting poor grammar or generally low-quality translation.

A translation can be grammatically perfect but culturally misleading. This is especially important for Indian languages, where meaning often depends on social relationships, respect, regional customs, festivals, food, and context.

If the proposed experiment succeeds, AI systems could use hidden-state analysis as an early warning system. Before showing a translation to a user, the system could identify cases where the language appears fluent but the cultural meaning may be incorrect.

The research is promising, but its main hypothesis has not yet been proven. The proposed dataset, human annotations, and model experiments still need to be completed.

This research asks one central question:

Can we detect a culturally wrong translation by looking inside an AI model, even when the translation sounds fluent and correct?

‍

‍