Fatwa Semantic Search

Project summary

Traditional keyword search often struggles with natural language queries, paraphrasing, and vocabulary differences. To improve retrieval quality, I designed a hybrid search pipeline that combines Gemini embeddings, BM25 retrieval, and Reciprocal Rank Fusion (RRF). The project also included evaluation workflows, human-labeling tools, and ranking experiments to measure and improve search relevance.

System view Sanitized

01 User query

->

02 Gibberish detection

->

03 Embedding retrieval

->

04 Keyword retrieval

->

05 Hybrid ranking

->

06 Validation Agent

->

07 Summarize selected fatwas

->

08 Return retrieved fatwas

Context

FatwaTok is an IslamWeb product delivered for the Qatar Ministry of Awqaf and Islamic Affairs. The wider IslamWeb platform receives an approved average of more than 10 million monthly visits, while this project’s retrieval corpus contains more than 165,000 fatwas. These figures represent platform reach and indexed content scale respectively.

The core question was: how do we retrieve the most relevant fatwa for a user query? Traditional keyword search is useful for exact terms, but it can miss relevant rulings when users phrase questions differently from the source wording. Semantic search improves meaning-based discovery, but it can miss precise lexical signals that matter in religious queries.

The resulting system combines both approaches. Gemini embeddings and BigQuery Vector Search support semantic retrieval, while BM25 preserves keyword-strength matching. Reciprocal Rank Fusion brings both result sets together into a single ranking that can be evaluated and improved.

Constraints

• Arabic user queries may be short, colloquial, or phrased differently from the source fatwa.
• Exact religious terms and named concepts still need strong keyword matching.
• Search quality needed measurable evaluation rather than relying on anecdotal checks.
• Human review was needed to label relevance and inspect difficult query cases.
• The retrieval system had to run in a production Google Cloud environment.

My role

I developed the retrieval system across embedding-based retrieval, BM25 keyword retrieval, hybrid ranking, evaluation workflows, human labeling support, LLM-assisted relevance review, and production deployment.

Retrieval design

The retrieval design uses two complementary paths. The semantic path generates Gemini embeddings for fatwa records and retrieves nearest neighbors through BigQuery Vector Search. The keyword path uses BM25 to preserve lexical relevance for exact terms, religious concepts, and names.

Results from both paths are merged with Reciprocal Rank Fusion. RRF was selected because it is simple to reason about, works well when combining heterogeneous rankers, and avoids forcing scores from different retrieval systems onto the same scale.

The public architecture intentionally omits proprietary weighting, query handling rules, and ranking thresholds.

Evaluation and safeguards

Search quality was evaluated through a combination of human labeling and LLM-assisted relevance review. Human labels provided grounded judgments for representative queries, while model-assisted evaluation helped expand review coverage and identify weak retrieval patterns.

The evaluation framework focused on whether the returned fatwas answered the user’s intent, not only whether they shared surface keywords. Exact benchmark values, labeled datasets, and acceptance thresholds remain confidential.

Production deployment

The retrieval system runs on Google Cloud, using BigQuery Vector Search for semantic retrieval and a keyword retrieval path for BM25 results. The serving layer exposes hybrid search behavior to the FatwaTok product experience.

The deployment keeps retrieval close to the broader IslamWeb data platform while allowing search ranking behavior to be evaluated and iterated independently.

Results and lessons

The delivered retrieval system made more than 165,000 Arabic fatwas searchable through hybrid semantic and keyword retrieval for FatwaTok on the IslamWeb platform, which averages more than 10 million monthly visits.

The central lesson was that production search quality depends on retrieval strategy and evaluation discipline. Embeddings are valuable, but real improvement comes from combining retrieval signals, reviewing failures, and building a repeatable relevance-evaluation loop.