mormot2-ai — the mormot.ai.* extension for mORMot2 (FPC · Delphi 7 … 13)

A mORMot-native AI extension for Free Pascal and Delphi 7 to 13 — the same compiler range as mORMot2. It covers the Model Context Protocol, LLM clients, embeddings, vector search and RAG, and it brings its own inference engine in plain Object Pascal: no Python, no llama.cpp, no foreign .so/.dll at run time. mORMot2’s HTTP stack, JSON, RTTI, SQLite and crypto are the foundation, not a dependency we tolerate.

Status, 2026-10-19: MCP, LLM clients, agent loop, RAG and the native vector store are done and in use. The native inference engine runs BGE-M3 embeddings from the same GGUF file as lembed, with the same quality and 1.6 to 2 times its speed on x86_64 (Linux / Windows); it passes all suites, its quality gate and a 90-minute soak test on Linux x86_64 too, and it is the recommended provider of the consuming backend. Three decoder families run behind ILlmClient - Qwen3, Qwen3-MoE (mixture of experts, Qwen3-30B-A3B) and Qwen2.5 - with greedy output in FP32 token-identical to transformers (for MoE on tiny reference models; the 30B model is checked against the CPU path): the agent loop calls tools with a model on this machine, greedy or sampled, with a prompt cache for the next turn. Weights come from safetensors or llama.cpp GGUF files (Q8_0, Q4_0, Q4_K_M, with the Q5_0 matrices llama.cpp writes for widths like Qwen2.5’s 896). The whole forward pass runs on the CPU (NEON, AVX2), on Vulkan (tested on an RTX 5090, an Adreno X1-85 and an M2 Pro through MoltenVK) and on CUDA. An RTX 5090 decodes Qwen3-0.6B at about 400 tokens/s (CUDA) and Qwen3-30B-A3B at about 97 (CUDA; Vulkan 90), choosing its experts on the GPU. Sliding-window attention and layers without rotary positions run on the CPU, Vulkan and CUDA (SmolLM3, so far without a chat template). gpt-oss-20b runs on the CPU from its published MXFP4 checkpoint or GGUF file (attention sinks, YaRN, biases, the clamped SwiGLU) and on Vulkan and CUDA with its MXFP4 experts (an RTX 5090 decodes it at about 125 tokens/s), and chats through ILlmClient in its Harmony format, tool calls included. A decoder also takes embedding rows instead of token ids and loads as one stack of a larger checkpoint - the talker and code predictor of Qwen3-TTS match PyTorch on the CPU, Vulkan and CUDA. Three test suites - about 7200 assertions on Windows x64, fewer where a feature is missing - green on every platform in the table below.

What it does