LogiMind

Multi-agent RAG system that answers natural-language questions about DHL’s public operational documents — rate guides, customs rules, packing requirements, prohibited items, incoterms, and sustainability reporting — or checks a simulated shipment’s tracking status. Out-of-scope questions are refused rather than answered from general knowledge: the system only speaks from what it actually retrieved.

Live demo: ui-production-b575.up.railway.app (rate-limited to 10 requests/minute)

Architecture

A one-time batch ingestion pipeline turns 14 PDFs (874 pages) into 5,133 chunks, embeds them, and writes them to both a vector store and a keyword index:

flowchart LR
    A[sources.py] --> B[downloader.py]
    B --> C["loader.py (PyMuPDF)"]
    C --> D[chunker.py]
    D --> E["embedder.py (OpenAI)"]
    E --> F[store.py]
    F --> G[(Qdrant)]
    F --> H[(SQLite)]

At query time, a fixed three-agent pipeline runs — no open-ended agent-to-agent conversation:

flowchart LR
    U[User] --> UI[Streamlit]
    UI --> API["FastAPI /query"]
    API --> O[Orchestrator]
    O --> P["PlannerAgent (Claude)"]
    P --> R["RetrieverAgent (deterministic)"]
    R -->|knowledge search| H[hybrid.py]
    H --> G[(Qdrant)]
    H --> S[(SQLite / BM25)]
    R -->|tracking lookup| T[tracking tool]
    R --> Resp["ResponseAgent (Claude)"]
    Resp --> API

Key decisions:

• Hybrid retrieval, not just one method. Qdrant vector search and BM25 keyword search run independently, get deduplicated, then re-ranked with a cross-encoder (ms-marco-MiniLM-L-6-v2) — the two methods surface genuinely different useful chunks on the same query.
• RetrieverAgent is deliberately not LLM-backed. PlannerAgent and ResponseAgent are Claude-backed AutoGen agents; the retriever is plain deterministic Python, since the planner’s decision already fully determines what needs to run.
• Fixed orchestration over AutoGen group chat. The orchestrator calls all three agents in a fixed sequence rather than an open-ended conversation, since there’s no genuine agent-to-agent negotiation to manage.
• Own evaluation loop instead of the ragas library. RAGAS’s metric definitions (faithfulness, answer relevancy) are reimplemented directly against the OpenAI SDK in ~30 lines, because ragas only imports through an old LangChain chain that conflicts with the numpy version required by sentence-transformers/scipy elsewhere in the stack.
• Two Docker images, not one. The API image carries the full ML stack (CPU-only torch build); the UI image is just an HTTP client (httpx + streamlit) — deployed as two separate services on Railway.

Results

Measured on 13 real queries through the full pipeline (Claude/Qdrant/OpenAI backends, the same ones the deployment uses), spanning every document category, a tracking lookup, a combined tracking+knowledge query, and an out-of-scope refusal:

Answer quality, scored by the eval loop above: 0.98 average faithfulness (answer claims checked against retrieved context, across the 11 queries with checkable claims) and 0.68 average answer relevancy across all 13. n=13 is enough to sanity-check the pipeline’s real cost/latency/quality profile, not a statistically rigorous benchmark.

Stack

Python 3.12, AutoGen, LangChain, Qdrant, BM25s, sentence-transformers, LangSmith, FastAPI, Streamlit, Docker, GitHub Actions, deployed on Railway.

Status

Ingestion, hybrid retrieval, the agent pipeline, monitoring (latency/cost tracking, faithfulness/relevancy evaluation), API/UI, Docker, CI, and deployment are all done — 108 tests, all external calls mocked. Still open: a curated ground-truth test set for retrieval-only quality scoring (context precision/recall).

Code: github.com/awais-de/logimind