Retrieval-augmented generation has become the default architecture for connecting large language models to external knowledge, but its reliability is still uneven in lower-resource, mixed-script, and technical domains. TekRAG-Persian is proposed as a data-centric benchmark for Persian technical question answering: a benchmark designed to measure how corpus cleaning, Persian-specific normalization, metadata enrichment, chunking, hybrid retrieval, and provenance-aware generation affect hallucination and citation quality. The core claim is deliberately narrow and testable: in Persian technical QA, hallucination is often as much a corpus-quality and evidence-representation problem as it is a model-quality problem.
Abstract
Retrieval-augmented generation, or RAG, improves language-model reliability by grounding generation in external documents. Yet most RAG evaluation remains concentrated in English or in broad-domain benchmarks. Persian technical writing creates a different reliability problem: it combines Persian orthography, zero-width non-joiner variation, Arabic-Persian codepoint drift, English acronyms, transliterated terminology, time-sensitive technology claims, and long-form editorial structure. This article introduces TekRAG-Persian, a research-grade benchmark design for hallucination-sensitive Persian technical question answering over a curated technology corpus. The benchmark compares raw, cleaned, metadata-enriched, and provenance-constrained corpus variants across sparse, dense, hybrid, and metadata-aware retrieval. It evaluates not only answer correctness, but also gold-support retrieval, citation precision, citation recall, atomic factual support, unsupported-claim rate, abstention behavior, calibration, and human judgments. The benchmark is positioned as a reproducible, data-centric study rather than a new base-model proposal. Its intended contribution is threefold: a Persian technical QA dataset with explicit evidence mapping, an evaluation protocol for RAG under mixed-script Persian conditions, and an error taxonomy for hallucination in Persian technical content.
Highlights
- Introduces TekRAG-Persian as a benchmark for Persian technical, attributed, hallucination-sensitive RAG.
- Frames hallucination reduction as a data-quality, retrieval, and provenance problem, not only a model-size problem.
- Defines raw, cleaned, metadata-enriched, and provenance-constrained corpus variants for controlled ablation.
- Specifies retrieval baselines including BM25, BM25F, multilingual dense retrieval, BGE-M3, Qwen3 Embedding, jina-embeddings-v3-style alternatives, hybrid fusion, and reranking, while separating retriever choice from current generator candidates such as GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, and Qwen3.6.
- Defines Persian-specific error categories: ZWNJ inconsistency, Arabic-Persian codepoint drift, mixed-script retrieval miss, terminology drift, and temporal hallucination.
- Provides an evaluation protocol aligned with benchmark papers: retrieval metrics, grounding metrics, hallucination metrics, abstention metrics, human evaluation, and statistical testing.
1. Introduction
Large language models are now used as interfaces to knowledge bases, documentation portals, customer-support repositories, news archives, and internal corporate memory. In these settings, a fluent answer is not enough. A useful system must retrieve the right evidence, cite the right source, refuse unsupported questions, and avoid turning partial evidence into confident invention. RAG was designed to reduce the distance between language-model generation and external knowledge, but the reliability of RAG depends on more than the generator. It depends on document extraction, text normalization, metadata design, chunk boundaries, retrieval method, reranking, prompt discipline, and evaluation.
Persian technical question answering exposes this dependency clearly. A Persian technical article may use “RAG” in Latin characters, “تولید تقویتشده با بازیابی” in Persian, “رگ” as an informal transliteration, and several inconsistent spellings for the same concept. It may contain English model names, Persian explanatory prose, product names, version numbers, dates, headings, code identifiers, and editorial summaries. A raw vector index may treat these forms as weakly related even when a human reader sees them as the same technical concept. The result is not simply lower retrieval recall; it is a higher probability that the generator will answer from an incomplete evidence set.
Existing Persian NLP work already includes valuable resources: broad Persian NLU benchmarks, Persian reading-comprehension datasets such as PQuAD and PersianQuAD, community QA resources, conversational QA, newer reasoning QA work such as PARSE, Persian language models, Persian instruction datasets, and Persian evaluation efforts. The defensible novelty claim for TekRAG-Persian is therefore not that Persian QA is absent. The defensible claim is narrower: there is still a need for an openly described benchmark focused on Persian technical, evidence-attributed, hallucination-sensitive RAG over a curated technology-news and analysis corpus. That is the gap this benchmark is designed to address.
The central research question is: How do corpus cleaning, Persian-specific normalization, metadata enrichment, chunking strategy, and provenance-aware generation affect retrieval accuracy, citation quality, and hallucination rates in Persian technical question answering? This question is practical enough to run, narrow enough to evaluate, and important enough to matter for real systems that serve Persian users.
2. Contributions
TekRAG-Persian is designed around three contributions. First, it defines a Persian technical QA benchmark built from curated technology articles with explicit question, answer, and evidence mapping. Second, it proposes a data-centric RAG evaluation protocol comparing raw, cleaned, metadata-enriched, and provenance-constrained corpora under the same retrieval and generation settings. Third, it introduces a hallucination and error-analysis taxonomy tailored to Persian technical content, including zero-width non-joiner variation, Arabic-Persian codepoint drift, mixed-script terminology, transliteration drift, temporal mismatch, and citation mismatch.
The paper deliberately avoids claiming a new foundation model. That is a strength, not a weakness. Many real RAG failures in production are caused by poor data representation, weak chunking, missing metadata, or insufficient evidence constraints. A benchmark that measures these effects can be more useful than another model-only comparison, especially in a lower-resource language where retriever and generator choices are already constrained by availability, cost, and multilingual competence.
3. Related Work and Research Gap
The literature behind TekRAG-Persian sits in four clusters. The first cluster is retrieval-augmented generation and information retrieval. The original RAG formulation established retrieve-then-generate as a way to ground language models in external knowledge. Dense Passage Retrieval showed that learned representations can outperform sparse methods in many open-domain QA settings. BM25 remains an essential lexical baseline, especially for acronyms, named entities, version strings, and code-like terms. BM25F extends sparse retrieval to structured fields and is especially relevant when titles, headings, summaries, tags, and dates can be weighted separately. Late-interaction and multi-vector retrievers such as ColBERT-style systems, BGE-M3, and newer multilingual embedding or reranking families such as Qwen3 Embedding and jina-embeddings-v3-style models offer stronger passage discrimination when first-stage retrieval is not enough.
The second cluster is hallucination, factuality, and attribution. Modern evaluation distinguishes between correctness, faithfulness to evidence, citation quality, atomic fact support, and answerability. FActScore-style atomic support is useful because a single fluent paragraph can contain both supported and unsupported statements. ALCE-style citation evaluation is useful because a citation is only valuable if it supports the claim attached to it. Attributed QA is useful because it frames question answering as an answer-plus-evidence task rather than a text-only generation task. RAGAS-style diagnostics are useful for fast inspection, but a serious benchmark should still include gold evidence and human evaluation.
The third cluster is data-centric AI. The benchmark’s thesis is aligned with the data-centric view: system quality is often improved by changing the data representation rather than only changing the model. For RAG, “data” includes more than raw text. It includes canonical URLs, article titles, section headings, dates, tags, summaries, author fields, entity aliases, normalized index text, original display text, chunk IDs, evidence spans, and provenance metadata. If these fields are missing or inconsistent, the generator may receive irrelevant or incomplete evidence even when the underlying corpus contains the answer.
The fourth cluster is Persian NLP and Persian QA. Persian already has meaningful benchmark work, including broad NLU resources, reading comprehension, conversational QA, community QA, reasoning QA, Persian encoder models, and Persian instruction data. PQuAD and PersianQuAD established large Persian reading-comprehension resources, while PARSE added a 2026 open-domain Persian reasoning QA benchmark. These resources strengthen the field, but they do not fully cover the combination targeted here: Persian technical domain content, mixed Persian-English terminology, long-form editorial articles, explicit citations, unanswerable questions, and hallucination-sensitive RAG evaluation. TekRAG-Persian should be positioned inside that gap, without overclaiming “first ever” status.
4. Corpus Profile
The intended source corpus is a Persian technical editorial knowledge base. Its domain profile includes artificial intelligence, data science, data governance, cybersecurity, software engineering, hardware and infrastructure, startup strategy, and future-technology analysis. This topic mix matters because it forces a retriever to handle both Persian explanations and English technical entities. A single article can include terms such as MCP, RAG, AI-Native IDEs, GPU, LLM, model collapse, vector database, semantic layer, prompt injection, and data provenance alongside Persian grammar and Persian punctuation.
The benchmark should report corpus statistics through a dataset card. Required fields include article count, token count, chunk count, category distribution, publication-date range, average article length, median chunk length, metadata completeness, language-script profile, alias dictionary size, crawl or export date, inclusion rules, exclusion rules, and release policy. If full article redistribution is not legally available, the benchmark can still release metadata, question records, evidence IDs, evaluation scripts, normalization rules, prompts, and reproducible instructions for rebuilding the corpus from authorized sources.
| Dataset-card field | Required reporting | Why it matters |
|---|---|---|
| Corpus source | Persian technical articles from a curated editorial corpus | Defines domain and licensing boundary |
| Temporal range | Start and end publication dates | Controls time-sensitive technical claims |
| Metadata completeness | Title, summary, category, tags, author, date, URL, headings | Determines whether field-aware retrieval is possible |
| Language profile | Persian text with mixed English technical terms | Motivates aliasing and mixed-script retrieval |
| Release status | Full text, partial text, metadata only, or scripts only | Determines reproducibility and legal safety |
5. Corpus Variants
The empirical centerpiece of TekRAG-Persian is a data-centric ablation. The same underlying articles should be indexed in multiple variants so that improvements can be attributed to data representation rather than uncontrolled model changes.
| Variant | Description | Research purpose |
|---|---|---|
| Raw | Minimal extraction from article HTML, weak cleanup, limited metadata, inconsistent chunk boundaries | Represents the common “dump the site into a vector database” baseline |
| Cleaned | Boilerplate removed, Unicode normalized, Persian characters standardized, headings preserved, duplicate whitespace removed | Tests whether basic Persian text hygiene improves retrieval and grounding |
| Metadata-enriched | Cleaned text plus title, summary, category, tags, author, date, canonical URL, heading path, alias list, and chunk position | Tests whether semantic and provenance structure improves evidence access |
| Provenance-constrained | Metadata-enriched retrieval plus generation prompts that require chunk citations and explicit refusal when evidence is insufficient | Tests whether evidence discipline reduces unsupported claims |
This design is important because it compares interventions that teams can actually deploy. Cleaning and metadata enrichment are cheaper than training a new model, easier to audit, and more stable across generator changes. If the benchmark shows measurable improvements from these interventions, the result is immediately useful for Persian search, support, documentation, and editorial QA systems.
6. Persian-Specific Normalization
Persian preprocessing is not a cosmetic step. It directly affects recall, evidence matching, and hallucination risk. The benchmark should preserve two representations: original_text for display and exact citation, and normalized_index_text for retrieval. The original text protects fidelity; the normalized text protects search.
| Issue | Example before | Example after | Retrieval impact |
|---|---|---|---|
| Arabic-Persian character drift | كاربرد RAG در سيستم هاي فارسي | کاربرد RAG در سیستمهای فارسی | Prevents false lexical mismatch |
| ZWNJ inconsistency | می شود / میشود | میشود | Improves tokenization and phrase matching |
| Mixed acronym forms | RAG / Rag / رگ / تولید تقویت شده با بازیابی | One alias group | Connects Persian and English queries |
| Boilerplate residue | Related articles, share links, menus | Removed | Reduces noisy chunks |
| Lost heading context | Body text without section label | Title plus heading path plus body | Improves metadata-aware retrieval |
The recommended cleaning pipeline is HTML extraction, boilerplate removal, Unicode normalization, Persian character mapping, zero-width non-joiner normalization, punctuation and whitespace cleanup, sentence segmentation, heading preservation, alias generation, deduplication, and metadata enrichment. Tools such as Hazm can provide normalization and sentence segmentation, but the benchmark should still document every custom rule because Persian technical terms often require domain-specific alias dictionaries.
7. Chunking Strategy
Chunking is a benchmark variable, not a background implementation detail. A fixed 400-token window may be convenient, but technical articles are usually organized by headings. Splitting across a heading boundary can remove the very signal that explains what a paragraph is about. Conversely, creating very large chunks can bury the evidence in the middle of a long context window and make citation less precise.
| Strategy | Default setting | Expected strength | Main risk |
|---|---|---|---|
| Fixed sliding window | 250, 400, or 600 tokens with small overlap | Cheap and reproducible baseline | Can split concepts and headings |
| Sentence-recursive | 4-8 sentence groups | Better local coherence | Still weak on section hierarchy |
| Heading-aware | H2/H3 sections, sub-split above 450 tokens | Strong interpretability and citations | Uneven chunk size |
| Semantic breakpoint | Embedding-based topic-shift boundaries | Topical purity | More compute and less deterministic |
| Late chunking | Embed longer context, then map to subspans | Context-preserving dense retrieval | Harder to reproduce |
The first public benchmark should include at least fixed-window, sentence-recursive, and heading-aware chunks. Semantic breakpoints and late chunking can be added as stronger ablations. For context assembly, the generator should usually receive the top three to eight high-value chunks, diversified across heading paths when needed, rather than a long list of weakly relevant snippets.
8. Question and Evidence Design
TekRAG-Persian should not be a trivia benchmark. It should represent the questions a technical reader actually asks: definitions, comparisons, procedures, causes, troubleshooting, multi-hop synthesis, temporal questions, and unanswerable prompts. Each question must map to minimal sufficient evidence, not merely to a source article.
| Question type | Target share | Example intent |
|---|---|---|
| Definition or concept explanation | 20% | What is retrieval-augmented generation? |
| Comparison or trade-off | 15% | When is BM25 stronger than dense retrieval? |
| Procedure or how-it-works | 15% | How does metadata-aware chunking change retrieval? |
| Cause or mechanism | 15% | Why can uncurated synthetic data cause model collapse? |
| Troubleshooting or failure mode | 10% | Why did the RAG system cite the wrong article? |
| Multi-hop synthesis | 15% | Combine evidence from two sections or articles |
| Temporal or version-sensitive | 5% | Answer based on publication date or product version |
| Unanswerable or adversarial | 5% | Refuse when the corpus does not support the claim |
A strong first release can use 600 questions: 360 train, 90 validation, 120 public test, and 30 hidden test. The hidden split is optional, but it helps prevent tuning directly to the public test set. The split should be article-aware, not purely random by question. All questions derived from the same source article should belong to the same supervision split when training or prompt development is involved.
Each QA record should include question ID, Persian question, optional English gloss, category, question type, difficulty, answerability, reasoning hops, gold answer, supporting document IDs, supporting chunk IDs, supporting spans, alias terms, and adjudication notes. Validation and test questions should be double annotated. Use Cohen’s kappa for binary and single-label decisions, weighted kappa for ordinal difficulty, and Krippendorff’s alpha when multiple annotators or multi-label evidence coding are involved.
9. Retrieval Methods
The retrieval comparison should be disciplined. The goal is not to create a retriever zoo; the goal is to test whether data quality and metadata improve evidence access across strong, recognizable baselines. The minimum comparison set should include BM25, multilingual dense retrieval, BGE-M3-style retrieval, hybrid reciprocal-rank fusion, BM25F with structured fields, and an optional reranker. A May 2026 implementation should treat BGE-M3 as a strong established baseline, then add at least one newer multilingual embedding or reranking family such as Qwen3 Embedding/Reranker or jina-embeddings-v3 if licensing, hardware, and Persian coverage fit the release constraints.
| Method family | Suggested setup | Reason to include |
|---|---|---|
| Sparse | BM25 with tuned k1 and b | Strong on acronyms, product names, version numbers, and exact technical terms |
| Field-aware sparse | BM25F with title, heading, summary, tags, body, and date fields | Directly tests metadata value |
| Dense | multilingual-E5-style or jina-embeddings-v3-style embeddings | Strong general multilingual semantic retrieval baseline |
| Unified multilingual | BGE-M3-style dense, sparse, or multi-vector modes | Flexible multilingual retrieval under one model family |
| Modern embedding plus reranking | Qwen3 Embedding/Reranker-style setup, or another documented 2025-2026 multilingual retriever | Keeps the benchmark current while preserving reproducible version pins |
| Hybrid | BM25 plus dense retrieval via reciprocal-rank fusion | Balances exact terms with semantic matching |
| Reranked | Retrieve top 50, rerank to top 5-10 | Improves precision before generation |
BM25 should not be treated as a weak baseline in this domain. Persian technical writing uses many exact identifiers: model names, acronyms, protocol names, CVE-like strings, version numbers, library names, and English product names. Dense retrieval may handle paraphrase better, but sparse retrieval can remain disproportionately strong on these exact-match signals. Hybrid retrieval is expected to be the most stable first-stage setup.
10. Generation and Provenance
Generator choice should be secondary to the data-centric thesis. A good experiment can use a fixed generator and vary corpus quality, then repeat the most important comparisons across one commercial model, one strong multilingual open or open-weight model, and one Persian-focused or Persian-capable baseline. As of May 2026, current documented candidates include OpenAI GPT-5.5, Anthropic Claude Opus 4.7, Google Gemini 3.1 Pro, and Qwen Qwen3.6. The benchmark should still pin exact model IDs, release notes, checkpoint names, context limits, inference stack, quantization settings, and run dates because API aliases and hosted model behavior can drift.
The provenance-aware prompt should require the model to answer in Persian, cite chunk IDs at sentence level, avoid uncited claims, and explicitly say that evidence is insufficient when retrieved chunks do not support the answer. The prompt should not reward longer answers. In technical QA, unsupported specificity is often worse than a short, cautious answer.
A strong RAG answer is not the longest answer. It is the shortest answer that correctly covers the question, cites the necessary evidence, and refuses unsupported additions.
The benchmark should include a no-RAG baseline. This is not because no-RAG is expected to win, but because it helps separate parametric memory from retrieval grounding. If a model answers correctly without retrieval but cannot cite evidence, the system may still fail the benchmark’s attribution requirement.
11. Evaluation Metrics
TekRAG-Persian should make three metric families primary: gold-support retrieval, atomic factual support, and citation quality. Surface similarity metrics such as ROUGE, BLEU, or token F1 can be reported for completeness, but they should not carry the paper. In hallucination-sensitive QA, a differently worded answer can be fully correct, and a fluent answer can be dangerously unsupported.
| Dimension | Metrics | Role |
|---|---|---|
| Retrieval | Recall@1/3/5/10, Precision@k, MRR, nDCG@10, gold-support hit rate | Measures whether evidence is accessible |
| Grounding | Sentence support rate, citation precision, citation recall, minimal-support recall | Measures whether claims are tied to evidence |
| Hallucination | Atomic factual support precision, unsupported-claim count, unsupported-claim rate | Measures invented or unsupported answer units |
| Abstention | Refusal precision, refusal recall, abstention F1, answerability accuracy | Measures behavior on unanswerable questions |
| Calibration | Expected calibration error, Brier score, confidence-correctness correlation | Measures whether confidence matches support |
| Human evaluation | Correctness, completeness, grounding, fluency, usefulness | Provides final judgment beyond automation |
Recommended headline metrics are gold-support hit rate, atomic support precision, and citation precision/recall. These metrics match the benchmark’s purpose: not merely answering, but answering with recoverable evidence.
12. Hypotheses
The benchmark should state hypotheses before experiments are run. This protects the work from post-hoc storytelling and makes the contribution easier for reviewers to evaluate.
- H1: The cleaned corpus will outperform the raw corpus on retrieval and grounding.
- H2: The metadata-enriched corpus will outperform the cleaned corpus on citation precision and multi-hop QA.
- H3: Hybrid retrieval will outperform sparse-only and dense-only retrieval overall.
- H4: BM25 will remain strong on acronym-heavy, code-mixed, and identifier-heavy questions.
- H5: Heading-aware chunking will outperform fixed-size chunking on technical explanatory questions.
- H6: Provenance-aware prompting will reduce unsupported claims, but may increase abstention and shorten answers.
- H7: Stronger generators will improve fluency and reasoning, but data-quality gains will remain visible when the generator is held fixed.
Statistical testing should use McNemar’s test for paired binary outcomes, paired bootstrap confidence intervals and Wilcoxon signed-rank tests for scalar per-question metrics, paired randomization or stratified bootstrap for retrieval metrics, and Holm-Bonferroni correction for multiple comparisons. Effect sizes and 95% confidence intervals should be reported alongside p-values.
13. Hallucination Taxonomy for Persian Technical QA
A generic hallucination label is too coarse for this benchmark. Reviewers, engineers, and dataset maintainers need to know why an answer failed. The taxonomy below combines general factuality problems with Persian-specific retrieval and normalization issues.
| Category | Description | Likely cause | Best mitigation |
|---|---|---|---|
| Unsupported elaboration | The answer adds details not present in retrieved evidence | Overconfident generation | Provenance-aware prompt and atomic support scoring |
| Citation mismatch | The cited chunk does not support the attached claim | Poor evidence selection or post-hoc citation | Reranking and citation audit |
| Terminology drift | A technical term is translated or conflated incorrectly | Bilingual jargon mismatch | Alias glossary and human review |
| Script normalization failure | Variant spellings prevent retrieval | Inconsistent Persian normalization | Canonical index text |
| Arabic-Persian codepoint drift | Arabic and Persian character variants are treated as different tokens | Unicode inconsistency | Character mapping before indexing |
| Mixed-script retrieval miss | The query uses Persian gloss while the article uses English acronym, or the reverse | Missing alias expansion | Mixed-script alias fields |
| Temporal hallucination | The answer ignores publication date or product version | Date metadata not used | Date filters and temporal QA labels |
| Cross-article causal leap | The answer combines true facts into an unsupported causal claim | Multi-hop overreach | Claim-level evidence evaluation |
| Boundary truncation | A chunk split removes a qualifier or condition | Bad chunking | Sentence-aware or heading-aware chunking |
| Over-refusal | The system refuses despite sufficient evidence | Overly cautious prompt or weak retrieval | Prompt tuning and retrieval diagnostics |
| Under-refusal | The system guesses when the corpus does not answer | Missing abstention policy | Explicit refusal branch and answerability labels |
14. Reproducibility Package
A Q1- or Scopus-oriented benchmark article must be reproducible. The submission package should include corpus construction rules, article inclusion/exclusion rules, normalization maps, alias dictionary logic, chunking scripts, retriever settings, prompt templates, train/validation/test IDs, evaluation scripts, annotation guidelines, inter-annotator agreement reports, and cost-latency logs.
| Artifact | Minimum content |
|---|---|
| Dataset card | Corpus source, license, version, fields, statistics, intended use, limitations |
| Annotation guidelines | Answerability rules, minimal-support definition, citation sufficiency rules |
| Prompt templates | Vanilla RAG, provenance-aware RAG, abstention prompt, no-RAG prompt |
| Hyperparameter grid | BM25, BM25F, dense model, top-k, reranker, chunk size, overlap |
| Evaluation scripts | Retrieval scoring, citation scoring, atomic support scoring, abstention scoring |
| Error examples | Representative Persian failures with labels, causes, and fixes |
If full-text release is restricted, the benchmark can still release question records, gold answers, evidence identifiers, metadata schemas, and code. That is less ideal than a fully open dataset, but it is more ethical and more reproducible than publishing unclear or unauthorized full-text dumps.
15. Ethics, Licensing, and Limitations
The first ethical issue is copyright and data release. Publicly visible articles are not automatically redistributable as a benchmark. The release policy should distinguish full text, derived chunks, metadata, questions, evidence IDs, and scripts. If rights are uncertain, the safest release is metadata plus annotations plus reproducible scripts for authorized corpus holders.
The second issue is benchmark leakage. Commercial and open models may have seen public technical web content during training. TekRAG-Persian should therefore include no-RAG baselines, require citations, and score grounding separately from answer correctness. Correct answers without evidence should not receive full credit in an attributed QA benchmark.
The third issue is linguistic fairness. Persian technical writing legitimately uses English terms, Persian equivalents, transliteration, and hybrid forms. Evaluation should not punish a system merely because it chooses an acceptable variant. It should punish unsupported claims, wrong citations, wrong concepts, and unjustified certainty.
The main limitation is domain scope. A benchmark built from technical editorial Persian may not transfer directly to legal, medical, religious, literary, or colloquial Persian. The benchmark also underrepresents multimodal evidence unless figures, charts, tables, and code blocks are explicitly modeled. These limitations should be stated plainly rather than hidden.
16. Submission Positioning for Q1 and Scopus Venues
The strongest academic positioning is not “Teknav wrote a successful article.” Search ranking and editorial popularity do not matter to journal peer review. The strongest positioning is: TekRAG-Persian is a data-centric, low-resource-language RAG evaluation study that measures how corpus quality, metadata, and provenance affect hallucination in Persian technical QA.
A journal submission version should be written in academic English, include a precise literature review, report dataset statistics, provide baseline comparisons, include confidence intervals, disclose limitations, and avoid unverifiable “first” claims. Candidate venue areas include information retrieval, applied artificial intelligence, data and knowledge engineering, language resources, and computer science applications. The final journal choice should be made after the abstract, results tables, and keywords are complete.
This Teknav publication establishes the benchmark design, research question, dataset schema, evaluation protocol, and novelty boundary. To convert it into a peer-reviewed Q1 or Scopus submission, the next required step is not more rhetoric; it is running the benchmark, filling result tables honestly, and releasing a reproducible artifact package.
17. Authorship, Editorial Credit, and Intellectual Contribution
This research concept is credited to Arsam Sabbagh, writer and AI/data researcher at Teknav. The article’s intellectual framing follows Arsam Sabbagh’s editorial line on data-centric AI, grounded generation, and Persian technical knowledge systems: the practical reliability of a model depends on the quality, structure, and traceability of the data it is asked to use. In a journal submission, the author contribution statement should be explicit rather than decorative. A suitable CRediT-style declaration would list Arsam Sabbagh for conceptualization, methodology, corpus design, benchmark framing, writing of the original draft, review and editing, and project administration. If additional annotators, engineers, or evaluators later contribute to dataset construction or experiments, their roles should be listed separately and accurately.
The article should also include a transparent drafting and editorial disclosure if required by the target venue. The safest disclosure is direct: any external assistance used for drafting, editing, formatting, or tooling should be named accurately, while research design, final claims, data validation, references, and responsibility for the manuscript remain with the named human author. Elsevier and Scopus-oriented venues increasingly expect this kind of transparency. The disclosure should not weaken the work if the empirical artifacts are real: a documented dataset, reproducible scripts, result tables, and human-reviewed annotation are the foundation of credibility.
The author biography inside the final manuscript should be concise and relevant. A good version is: Arsam Sabbagh is an AI and data engineering researcher focused on retrieval-augmented generation, data-centric AI, Persian technical content systems, and evaluation of grounded language-model applications. This connects the author to the topic without overstating credentials or inventing institutional affiliations. A Q1 or Scopus submission should never fabricate university affiliation, funding, laboratory membership, or reviewer relationships. If the author is independent or affiliated with Teknav, that should be stated plainly.
18. Proposed Experimental Result Tables
The public Teknav article can define the invention and benchmark, but the journal version must include measured results. The results section should be planned before experiments are run so that the final paper does not become selective or biased. The first table should report corpus statistics and dataset composition. The second should compare corpus variants while holding retriever and generator constant. The third should compare retrieval families while holding corpus variant constant. The fourth should report generation and grounding results. The fifth should show error categories and representative examples. These tables should be compact enough for a main manuscript, with extended per-category and per-difficulty results placed in an appendix.
| Table | Rows | Columns | Reviewer question answered |
|---|---|---|---|
| Corpus statistics | Raw, cleaned, enriched, provenance-constrained | Articles, tokens, chunks, median chunk length, metadata completeness | Is the dataset real and documented? |
| Corpus ablation | Raw, cleaned, enriched, provenance-constrained | Recall@5, MRR, citation precision, atomic support precision, unsupported-claim rate | Does data quality actually help? |
| Retriever comparison | BM25, BM25F, dense, BGE-M3, Qwen3 Embedding, hybrid, reranked hybrid | Recall@1/5/10, nDCG@10, support hit rate, latency | Which retrieval family is strongest? |
| Generator comparison | No-RAG, vanilla RAG, provenance-aware RAG, rerank-RAG | Correctness, grounding, abstention F1, unsupported claims | Does citation discipline reduce hallucination? |
| Error taxonomy | Failure categories | Frequency, severity, typical cause, recommended mitigation | What exactly fails in Persian technical RAG? |
The result tables should not be filled with estimated numbers. The strongest version of this article will be one that reports modest but trustworthy measurements. If the cleaned corpus improves Recall@5 by a small but consistent margin, that is still valuable. If metadata improves citation precision only for multi-hop questions, that is a more interesting finding than a broad exaggerated claim. If provenance-aware prompting reduces unsupported claims but increases refusals, that trade-off should be reported rather than hidden. Q1 reviewers generally prefer a precise, limited result over a dramatic but weakly supported story.
19. Statistical Analysis Plan
The benchmark should treat each question as a paired evaluation unit. If the same question is answered by raw-RAG, cleaned-RAG, metadata-RAG, and provenance-aware RAG, then the comparison is paired by design. This allows stronger statistical tests and reduces noise from question difficulty. For binary outcomes such as “gold support retrieved” or “answer contains unsupported claim,” use McNemar’s test. For scalar outcomes such as citation precision, atomic support precision, MRR, or latency, report paired bootstrap confidence intervals and Wilcoxon signed-rank tests. For rank-based retrieval metrics, use paired randomization or stratified bootstrap. For human ordinal ratings, use Wilcoxon signed-rank tests or a cumulative-link mixed model with system and question type as factors.
Multiple comparisons should be controlled. The paper can pre-register a small set of primary comparisons: cleaned versus raw, metadata-enriched versus cleaned, hybrid versus dense-only, and provenance-aware generation versus vanilla RAG. Secondary comparisons, such as late chunking or HyDE-style query expansion, can be marked as exploratory. Holm-Bonferroni correction is a reasonable default because it is less conservative than a plain Bonferroni correction while still controlling family-wise error. Every main table should include effect sizes and 95% confidence intervals, not only p-values.
The benchmark should also report practical significance. A statistically significant two-point improvement in a metric may not matter if it doubles latency or makes the system too expensive. Conversely, a moderate increase in abstention may be acceptable if unsupported claims fall sharply. The final discussion should therefore combine statistical significance, effect size, cost, latency, and human usefulness. That is the kind of analysis that makes the study useful to both researchers and practitioners.
20. Dataset Release and Governance Model
A credible benchmark needs governance. TekRAG-Persian should use semantic versioning: for example, v0.1 for the internal pilot, v1.0 for the first public release, and later versions for new topics, new temporal slices, or corrected annotations. Every version should preserve immutable question IDs and evidence IDs. If a question is corrected, the old version should remain traceable so that prior results can be interpreted.
The release package should separate four layers: corpus metadata, question annotations, evidence references, and full text. If full article text can be released, include it with a clear license. If it cannot be released, publish the metadata schema, question set, gold answers, evidence IDs, canonical URLs, chunking rules, and scripts that allow authorized users to rebuild the benchmark. This is a common compromise for benchmarks built from copyrighted or publisher-owned text. It is weaker than a fully open corpus, but it is still scientifically useful if the reconstruction process is deterministic and the evaluation scripts are public.
Hidden test governance should be simple. Keep a small hidden set controlled by Teknav or a neutral maintainer. Require systems to submit outputs, not code execution on private servers, unless infrastructure is available. Publish aggregate results and confidence intervals, but do not leak hidden labels. If the project grows, a public leaderboard can be added, but it should not become the main scientific contribution. The benchmark’s real value is the dataset design, evidence mapping, and Persian-specific hallucination analysis.
21. Reference Implementation Blueprint
The reference implementation should be intentionally boring. It should prefer standard tools over clever one-off code. A strong baseline stack is: a corpus exporter, an HTML cleaner, a Persian normalizer, a chunk builder, a metadata serializer, a sparse index, a dense vector index, a hybrid fusion layer, a reranker, a prompt runner, and an evaluator. The pipeline should be runnable from a single configuration file that fixes corpus version, chunking strategy, retriever, generator, top-k, prompt template, and output directory.
benchmark:
name: TekRAG-Persian
corpus_version: v1.0
split: test
corpus_variant: metadata_enriched
chunking: heading_aware
retriever:
sparse: bm25f
dense: bge_m3
fusion: reciprocal_rank_fusion
top_k: 8
generator:
mode: provenance_aware_rag
temperature: 0
cite_chunk_ids: true
refuse_without_support: true
evaluation:
retrieval: [recall_at_5, mrr, ndcg_at_10]
grounding: [citation_precision, citation_recall, atomic_support_precision]
abstention: [refusal_precision, refusal_recall, abstention_f1]
The reference implementation should log every retrieved chunk and every final answer. This is essential for debugging. A result row without the retrieved evidence is almost useless because it cannot explain whether the failure came from retrieval, reranking, prompting, or generation. Logs should include query ID, normalized query, retrieved chunk IDs, scores, reranker scores, prompt hash, generator version, answer text, citations, evaluation labels, and runtime. If commercial APIs are used, store model name and run date. If open models are used, store checkpoint hash or model card version.
22. Linked Reference Base
The article’s research base should use primary sources where possible. The following links are included as a starting reference base for the web article and as a checklist for the journal manuscript bibliography. The final submission should convert them into the citation style required by the target venue.
| Area | Reference | Use in TekRAG-Persian |
|---|---|---|
| RAG | Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks | Foundational retrieve-then-generate framing |
| Dense retrieval | Dense Passage Retrieval for Open-Domain Question Answering | Dense retriever baseline and contrast with sparse retrieval |
| Citation evaluation | Enabling Large Language Models to Generate Text with Citations | ALCE-style citation quality and evidence-grounded evaluation |
| Atomic factuality | FActScore | Atomic support precision and unsupported-claim analysis |
| RAG evaluation | RAGAS | Reference-free diagnostics for RAG pipelines |
| Multilingual retrieval | Multilingual E5 Text Embeddings | General multilingual dense retrieval baseline |
| Multilingual retrieval | BGE-M3 Embedding | Multilingual, multi-function retrieval baseline |
| Multilingual retrieval | Qwen3 Embedding | Modern multilingual embedding and reranking candidate |
| Multilingual retrieval | jina-embeddings-v3 | Task-adapted multilingual embedding candidate |
| Persian QA | PQuAD | Persian reading-comprehension benchmark context |
| Persian reasoning QA | PARSE | Recent Persian reasoning QA benchmark context |
| Commercial generator | OpenAI model documentation | GPT-5.5 model ID, context window, tool support, and snapshot-pinning practice |
| Commercial generator | Claude Opus 4.7 API documentation | Claude Opus 4.7 model ID, context window, and versioning notes |
| Commercial generator | Google Gemini 3.1 Pro announcement | Gemini 3.1 Pro availability and preview status for API and Vertex AI users |
| Open-weight generator | Qwen3.6 repository | Qwen3.6 model family, open-weight release notes, and exact checkpoint selection |
| Persian preprocessing | Hazm Persian NLP toolkit | Normalization, tokenization, and preprocessing baseline |
| Publishing pathway | Elsevier guide: publish in a journal | Submission workflow and research-publication expectations |
| Indexing criteria | Scopus content policy and selection | Peer review, ethics, international readability, and journal quality criteria |
23. Frequently Asked Questions
The following questions clarify the intended scope for readers, search systems, and future reviewers. They are included because TekRAG-Persian should be understood as a benchmark protocol and reproducibility plan, not as a finished claim that all experiments have already been run.
What is TekRAG-Persian?
TekRAG-Persian is a proposed benchmark and evaluation protocol for Persian technical retrieval-augmented generation. It focuses on whether systems retrieve the right evidence, cite the right source, avoid unsupported claims, and refuse questions that cannot be answered from the corpus. Its central object is not a chatbot interface; it is a reproducible way to test Persian technical QA under controlled corpus and retrieval conditions.
Is TekRAG-Persian a new language model?
No. The benchmark is intentionally data-centric. It does not claim that a new base model is required. Instead, it asks whether Persian-specific normalization, metadata enrichment, chunking, hybrid retrieval, reranking, and provenance-aware prompting can measurably reduce hallucination and improve citation grounding when the generator is held constant.
Why does Persian RAG need a separate benchmark?
Persian technical content has retrieval problems that are easy to miss in English-only evaluations: zero-width non-joiner variation, Arabic-Persian codepoint drift, English acronyms inside Persian prose, transliterated technical terms, product names, version numbers, and date-sensitive claims. These conditions affect whether the right evidence is retrieved and whether a generated answer can be trusted.
What must be done before a journal submission?
The protocol must be converted into empirical results. A submission-ready version should include the dataset card, annotation guidelines, corpus statistics, baseline comparisons, confidence intervals, evaluation scripts, and a transparent data availability statement. The web article defines the invention and research plan; the journal paper must add measured evidence.
24. Search Scope and Reader Fit
This article is intended for readers searching for a Persian RAG benchmark, a Persian NLP benchmark, a Persian technical question answering benchmark, or a reproducible method for RAG hallucination evaluation. It is also relevant to teams working on citation-grounded RAG, evidence-attributed QA, hybrid retrieval for low-resource languages, and data-centric AI evaluation. The focus is deliberately narrower than general Persian language understanding: TekRAG-Persian targets technical articles, mixed Persian-English terminology, citation quality, answerability, and provenance-aware generation.
The practical search intent behind the work is clear: researchers and engineers need to know whether cleaning a Persian corpus, preserving headings, adding metadata, using BM25F or hybrid retrieval, and forcing chunk-level citations can reduce unsupported claims. The benchmark is therefore useful to information retrieval researchers, NLP benchmark builders, RAG engineers, data quality teams, and journal reviewers evaluating low-resource language AI systems.
25. What Makes the Work an Invention Rather Than an Essay
The inventive part of TekRAG-Persian is not the phrase “Persian RAG.” It is the combination of benchmark scope, corpus variants, evidence attribution, Persian-specific normalization, mixed-script aliasing, and hallucination taxonomy. A broad essay says that Persian RAG is important. TekRAG-Persian specifies how to measure it. A broad essay says that data quality matters. TekRAG-Persian defines raw, cleaned, metadata-enriched, and provenance-constrained variants and asks whether each intervention changes retrieval, citation, abstention, and hallucination metrics. A broad essay says models hallucinate. TekRAG-Persian breaks hallucination into operational categories that can be counted, compared, and mitigated.
For this reason, the strongest title remains benchmark-oriented: TekRAG-Persian: A Data-Centric Benchmark for Reducing Hallucination in Persian Technical Question Answering. The title signals artifact, method, language, domain, and evaluation target. It is more publishable than a trend title because it promises a measurable contribution. It is also realistic: the work does not need to invent a new model architecture to be useful. It needs to prove that careful data engineering and evidence discipline can produce measurable reliability gains in a Persian technical domain.
The next milestone is an empirical pilot. A credible pilot can start with 100 articles, 150 questions, three corpus variants, BM25, one dense retriever, one hybrid retriever, and one fixed generator. If the pilot validates the pipeline, the full study can expand to 300-600 questions and a stronger model comparison. This staged approach reduces risk and makes the research easier to debug. It also gives the author a defensible development story for reviewers: the benchmark was not written as a single speculative essay; it was iterated from a pilot into a reproducible evaluation framework.
26. Declarations for a Journal-Ready Manuscript
The manuscript should include a complete declarations section before submission. A suitable data availability statement would say that the benchmark will release question annotations, metadata schema, evidence identifiers, prompts, evaluation scripts, and corpus reconstruction instructions, while full text release will depend on copyright and licensing clearance. A suitable code availability statement would point to the public repository containing preprocessing scripts, chunking rules, retrieval configuration, evaluation code, and result-table generation. A suitable conflict of interest statement would disclose that the source corpus is associated with Teknav and that Arsam Sabbagh is credited as the author and research designer. This is not a disqualifying conflict if it is disclosed; it simply means the evaluation protocol must be auditable.
The ethics statement should explain that the benchmark uses technical editorial text and does not intentionally collect private personal data. Annotators should be told not to add private information to gold answers. If user questions are collected in a later phase, they should be anonymized or excluded unless consent and privacy review is in place. The funding statement should not invent support. If the work is self-funded or internally supported by Teknav, say so. The authorship and tooling disclosure should state exactly what assistance was used and confirm that the author reviewed, corrected, and accepts responsibility for the final manuscript.
The manuscript should also include a limitations of claim paragraph. TekRAG-Persian is not a general benchmark for all Persian language understanding, all Persian dialects, or all professional domains. It is a benchmark for Persian technical, editorial, citation-grounded QA. Its value comes from precision. The narrower scope makes the work more testable, easier to reproduce, and more defensible under peer review. That is exactly the posture a Q1 or Scopus-facing article should take: rigorous, transparent, limited where appropriate, and useful to future researchers.
Conclusion
TekRAG-Persian reframes hallucination in Persian technical QA as a measurable systems problem. The benchmark asks whether cleaned text, Persian-aware normalization, heading preservation, metadata enrichment, hybrid retrieval, and provenance-aware generation can reduce unsupported claims without requiring a new foundation model. This is a practical and academically defensible contribution because it focuses on a real gap: attributed, hallucination-sensitive RAG for Persian technical content.
The central lesson is simple: a RAG system does not become trustworthy because it uses a powerful generator. It becomes trustworthy when the corpus is clean, the evidence is findable, the metadata is meaningful, the citations are checked, and the model is forced to respect what the retrieved documents actually support. For Persian technical knowledge systems, that is the foundation on which reliable AI should be built.
The current version is a benchmark protocol and research design. It deliberately avoids fabricated performance numbers; numerical tables should be completed only after the benchmark is executed and the evaluation artifacts are available for review.
پرسشهای پرتکرار
What is TekRAG-Persian?
TekRAG-Persian is a proposed benchmark and evaluation protocol for Persian technical retrieval-augmented generation, focused on hallucination, citation grounding, and corpus quality.
Is TekRAG-Persian a new language model?
No. The article positions TekRAG-Persian as a data-centric benchmark, not a base model. It evaluates how corpus cleaning, metadata, retrieval, and provenance rules affect grounded QA.
Why does Persian RAG need a separate benchmark?
Persian technical content combines Persian orthography, ZWNJ variation, Arabic-Persian codepoint drift, English acronyms, transliteration, and date-sensitive technology claims, which make retrieval and citation evaluation different from English-only QA.
What must be done before journal submission?
The benchmark design must be converted into empirical experiments with released annotation rules, result tables, confidence intervals, evaluation scripts, and transparent data availability statements.