iHorizons · 2025 - Present
Fatwa Content Processing Pipeline
A large-scale content-processing system responsible for keeping the IslamWeb fatwa corpus synchronized, enriched, searchable, and audio-ready through automated daily workflows.
- Role
- Senior AI Engineer
- Client
- Qatar Ministry of Awqaf and Islamic Affairs
- Scale
- 165K+ Arabic religious documents
- Focus
- Embedding · Content enrichment · Batch inference
Project summary
The pipeline ensures that newly published fatwas automatically move from raw content to production-ready assets. Each daily run performs content synchronizationn, embedding, metadata generation, classification, search-artifact creation, and Quran-aware text-to-speech processing, providing fresh content for search, answer generation, and audio experiences across IslamWeb services.
Context
IslamWeb’s fatwa corpus is continuously growing as new religious rulings are published. These documents must be synchronized, enriched, classified, indexed, and prepared for downstream services without requiring manual intervention.
Unlike a typical content-processing pipeline, this corpus contains Arabic religious content where accuracy and faithful representation are particularly important. Generated summaries must remain grounded in the source fatwas, content classifications must be reliable, and audio experiences require special handling for Quranic text to avoid treating Quran verses as ordinary prose during text-to-speech generation.
The scale of the corpus introduces additional operational challenges. Enrichment, classification, and audio-generation workflows must process large volumes of content through batch inference pipelines while handling retries, failures, and external service rate limits.
The resulting solution is a production content-processing pipeline responsible for keeping more than 165,000 fatwas synchronized, enriched, searchable, classified, and audio-ready across the broader IslamWeb ecosystem.
Constraints
- New fatwas needed to be detected and synchronized automatically from source systems.
- Large-scale Arabic content required reliable batch processing while respecting external service limits and operational quotas.
- Generated summaries needed to remain faithful to the source fatwas and avoid introducing unsupported religious guidance.
- Quran verses required dedicated detection and handling before text-to-speech generation.
- Some fatwas contain sensitive content that required automated classification workflows to support age-appropriate user experiences.
- Search indexes and downstream services needed to remain synchronized with newly published content.
- The pipeline needed to operate reliably on a recurring schedule with minimal manual intervention.
My role
I developed the end-to-end content-processing pipeline, including content synchronization, batch enrichment workflows, metadata generation, content classification, Arabic text-to-speech generation, Quran-aware processing, BM25 artifact generation, scheduling, monitoring, and production deployment.
Pipeline Design
The pipeline runs on a scheduled basis and continuously scans for newly published fatwas. New content is detected, fetched, processed, and propagated to downstream systems through a series of independent processing stages.
Content Synchronization
The synchronization workflow identifies newly published fatwas and retrieves the required source content. This allows downstream services to remain aligned with the latest corpus without requiring manual updates.
The workflow also manages content updates and publication readiness, ensuring that newly available fatwas can flow through enrichment and indexing stages automatically.
Content Enrichment
Newly synchronized fatwas are enriched through large-scale batch inference workflows that generate summaries, metadata, and additional content attributes used by downstream applications.
Batch processing was designed to handle large volumes of Arabic content efficiently while providing retry behavior, failure recovery, and operational safeguards around long-running inference jobs.
Classification
The pipeline includes automated classification workflows used to identify specific content categories.
Examples include sensitive-content classification and modern-fatwa classification, allowing downstream products to apply category-aware behavior without requiring manual review of the entire corpus.
These classifications help support safer user experiences and improve content discoverability across the broader platform.
Quran-Aware Audio Generation
The pipeline prepares Arabic religious content for text-to-speech generation.
Special handling is applied to Quran-related text before audio generation. Quran verses are detected and transformed into appropriate references when necessary, while Arabic text preparation workflows improve pronunciation quality and downstream audio generation.
The resulting workflow generates production-ready Arabic audio assets suitable for long-form religious content.
Search Artifact Generation
The pipeline generates and publishes BM25 retrieval artifacts used by keyword-search systems.
By rebuilding and publishing search artifacts as part of the synchronization workflow, retrieval systems remain aligned with the latest processed corpus and newly published fatwas become searchable without manual intervention.
Quality and Reliability
The pipeline was designed around operational reliability rather than a single model component.
Individual processing stages can be executed, monitored, retried, and validated independently. This separation makes it possible to improve enrichment, classification, audio generation, and search-index workflows without affecting unrelated parts of the system.
Production safeguards focus on processing correctness, retry behavior, artifact consistency, successful publication of downstream outputs, and reliable handling of sensitive religious content.
Observability and Production Deployment
The pipeline runs as a scheduled production workflow on Google Cloud.
Cloud Scheduler triggers recurring execution, while processing services generate artifacts, update storage systems, and publish outputs for downstream consumers.
Structured logging and operational dashboards provide visibility into pipeline execution, processing status, failures, and synchronization activity. This allows recurring content operations to run automatically while keeping failures visible and recoverable.
Results and Lessons
The pipeline continuously processes and enriches a corpus of more than 165,000 Arabic religious documents, supporting synchronization, summarization, classification, audio generation, and search-index updates across IslamWeb services.
One of the key lessons was that production AI systems depend heavily on workflow engineering around the models themselves. Scheduling, retries, monitoring, artifact publication, Arabic-language processing, and operational reliability were as important as the enrichment models used within the pipeline.
The project demonstrated that large-scale AI content pipelines require equal attention to model quality, operational robustness, and domain-specific safeguards in order to remain reliable in production.