At Serideo AI, we treat benchmarking as a research exercise, not a marketing one. This report documents the methodology, dataset, evaluation process, and results from our most recent benchmarking run

1. Objective

Internal quality metrics tell us how consistent our own annotators are with each other. They don't tell us how our pipeline performs against an external, independently verified standard. This run was designed to close that gap: to measure our production pipeline's real-world accuracy on two publicly available benchmarks covering standard Mandarin classification and code-switched, dialect-heavy Mandarin NER.

2. Methodology

Dataset size. The evaluation covered 5,000 tasks in total, sampled across the two benchmarks. This is large enough to produce a statistically meaningful accuracy figure per benchmark, rather than a small pilot-scale estimate.

Annotation and classification process. Every task was processed through the same AI-assisted annotation workflow we are building into our web platform not a separate, benchmark-only setup. Concretely:

• An initial pass used AI-assisted classification to generate a first-draft label for each task.
• Human annotators then manually reviewed, corrected, or confirmed each AI-assisted output. No task was scored purely on the model's first pass every single one went through manual review before being finalized.
• For scoring against ground truth, we used external open-source evaluation tooling (rather than an in-house scorer), so that our accuracy/precision numbers are computed the same way any other team using the same public benchmark would compute them. This keeps the results directly comparable to other reported scores on the same benchmarks.

Team involved. A team of five or more spanning annotators and reviewers worked on this run, following our standard escalation process for ambiguous or borderline cases (common in code-switched text, where a single sentence can plausibly take two different labels depending on which language's convention is applied).

Why this setup matters methodologically: because the same AI-assisted + manual pipeline that ran this benchmark is the one going into production, the resulting scores are a direct, honest read of what a customer using Serideo today would actually get not a tuned or cherry-picked best case. evaluating our multilingual text classification and annotation pipeline against two public benchmarks.

3. Benchmarks and Results

3.1 HKCanto-Eval (Mandarin)

Task: Mandarin-language text classification. Result: 65% accuracy across the sampled task set.

This is a first-pass baseline under our standard (non-benchmark-tuned) pipeline. Manual error review by our team identified that most misclassifications clustered around informal register and short, ambiguous text fragments a specific, addressable gap rather than a broad weakness.

3.2 Code-Switching & Dialect-Speech Benchmark (Singapore)

Task: Mandarin Named Entity Recognition (NER) in code-switched, dialect-rich speech (Singaporean Mandarin mixed with English and regional terms). Result: 87.93% accuracy/precision.

This is a meaningfully stronger result, and it lines up with where our pipeline is purpose-built to perform: mixed-language, dialect-heavy content the exact category our annotation guidelines and reviewer training are designed around.

more info - https://research-serideo.framer.website/

4. Findings

1. The AI-assisted + manual review pipeline holds up under external, independently-scored evaluation not just internal QA. Using external open-source scoring tools rather than an in-house one was a deliberate methodological choice to keep the results credible and comparable.
2. Performance is strongest exactly where our pipeline is optimized for — code-switched, dialect-rich Mandarin. The near-88% NER score reflects annotation guidelines and reviewer calibration that were built specifically around mixed-language content, not adapted from a generic single-language pipeline.
3. The 65% score on standard Mandarin classification is a diagnostic result, not a discouraging one. Because every task went through manual review, we have direct visibility into where the errors came from — concentrated in informal register and short ambiguous text giving us a specific, addressable target for the next guideline revision.
4. Team-scale involvement (5+ annotators/reviewers) across 5,000 tasks means these numbers reflect a real operational run, not a small proof-of-concept.

Conclusion

The benchmark evaluations indicate that Serideo AI's multilingual annotation and quality assurance pipeline is capable of producing high-quality datasets across challenging multilingual and code-switched tasks.

Our evaluations on HKCanto-Eval (Mandarin) and the Code-Switching & Dialect-Speech Benchmark (Singapore) demonstrate that a structured human-in-the-loop workflow, supported by multi-layer quality assurance, can achieve consistent performance on complex language understanding tasks. These results also validate the effectiveness of our annotation methodology for multilingual AI data creation.

While benchmark scores provide an important measure of technical performance, they represent only one aspect of dataset quality. We view these evaluations as continuous feedback mechanisms that help identify improvement opportunities, refine annotation guidelines, and strengthen our quality assurance processes.

Going forward, Serideo AI will continue expanding benchmark evaluations across additional languages, domains, and AI tasks while further enhancing annotation quality, consistency, and scalability. Our objective is to build reliable, enterprise-grade multilingual datasets that enable the development of more accurate, robust, and inclusive AI systems.

see report - https://lnkd.in/dZF8Z-b5

‍