EFQM/RADAR AI Report Analyzer

LangGraph coordinates parallel criterion agents and carries the assessment state between them. Pydantic v2 defines the contracts they exchange, so malformed or incomplete output is rejected before it can reach the report stage. Qdrant retrieval is filtered by assessment item, which keeps guidance and source evidence tied to the question being reviewed.

The runtime stays local by default. Ollama serves Qwen2.5 14B, while nomic-embed-text produces embeddings. PyPDF handles normal extraction, and RapidOCR with ONNX Runtime recovers text from scanned pages and chart content. FastAPI and Jinja2 provide the upload and assessor-review interface.

The CI release gate checks retrieval precision, recall, and F1, citation coverage, schema validity, question coverage, report completeness, evidence validity, and three-run scoring stability. Each run records its model and execution configuration with a SHA-256 digest, then writes machine-readable metrics to a build artifact and a pull-request summary.

One evidence calibration covered 573 segments and all 38 assessment-guide items. That run recorded zero retrieval or grounding failures, 100 percent schema validity, 100 percent question coverage, and 100 percent evidence-reference validity. The report rendered 84 reasoning records and exported 143 supporting quotes without an unmatched quote.

A corrected PDF-ingestion run routed all 65 Results records: 42 were accepted through OCR and 23 were flagged for review. None were silently lost or attached to the wrong item. A synthetic retrieval fixture reached F1 1.0, which I treat as an engineering regression result rather than proof of real-world assessment accuracy.

The assessment guide is the scoring authority available to the pipeline. EFQM and RADAR support evidence analysis and report structure, but the software does not fill in unavailable official values with convenient defaults. If an authoritative weight, band, or finalisation rule is missing, official scoring stays blocked and any supported interim output is labelled provisional.

That boundary also limits what the report claims. The system performs document-based desk assessment. It does not claim interviews, site visits, assessor consensus, jury decisions, or external verification. Unsupported claims become gaps or questions for a human assessor, not facts generated by the model.

I lead a five-person team whose members began without prior experience in RAG, agent orchestration, structured LLM output, or EFQM/RADAR assessment. I built a seven-day onboarding programme with practical exercises and minimum pass conditions, then divided the system into workstreams for ingestion, evidence governance, retrieval evaluation, review interfaces, reporting, and quality.

Each workstream has an owner, dependencies, acceptance criteria, and weekly deliverables. OpenSpec documents and GitHub issues make the contracts explicit. Protected-branch pull requests must pass tests, Ruff, mypy, security checks, and the project quality gates before integration.

The team-integrated build reached 303 tests at 89.94 percent coverage. A later autonomous implementation expanded the suite to 1,360 tests at 87.48 percent coverage. Those numbers describe engineering coverage, not model accuracy.

The pipeline is already usable for governed desk-assessment drafts, but the remaining work is about defensibility rather than adding more agents. Official scoring and finalisation stay blocked until the missing authoritative scoring configuration is available.

These are optional directions, not committed deliverables. They only make sense after the current governance and evaluation gaps are closed.