Revolutionizing AI Inference: ZML's Free Software for Faster Performance

TL;DR

• French startup ZML has released ZML/LLMD, a free (though not open-source) inference server that accelerates AI performance across diverse chips including Nvidia, AMD, Google TPU, Apple Metal, and Intel Arc.
• The software, endorsed by Turing Award winner Yann LeCun, aims to eliminate hardware silos and reduce operational costs for enterprises by enabling mixed-chip usage without separate optimization work.
• ZML/LLMD is currently a technical preview in alpha with a compact 2.4GB container, written in Zig, but still limited to single-GPU operation and specific model families like Llama and Qwen.

Revolutionizing AI Inference: ZML's Bold New Free Tool

The AI infrastructure landscape is witnessing a significant shift as ZML, a Paris-based startup, has launched its latest innovation: ZML/LLMD. This newly released inference server is designed to shatter the existing "silos" that have traditionally forced AI developers to optimize models separately for every chip architecture. By allowing open-source large language models to run at peak speed across a variety of hardware—including Nvidia GPUs, AMD processors, Google's TPU, Apple Metal, and Intel Arc—ZML is positioning itself as a critical player in democratizing high-performance AI inference.

The company's ambition is clear: to make different chips available for AI use cases at their maximum available speed, and sometimes even faster, according to ZML founder Steeve Morin. Unlike ZML's previous public project, an ML framework released in 2024, ZML/LLMD is not open-source. However, it is launching as a completely free product with the strategic goal of learning about usage patterns and building a community around the technology.

Endorsement from a Turing Award Legend

The credibility of ZML's new tool is bolstered by a high-profile endorsement from Yann LeCun, the renowned Turing Award winner and former chief AI scientist at Facebook. LeCun's support signals that ZML/LLMD addresses one of the industry's most pressing problems: the astronomical cost of AI inference.

With LeCun's backing, the Paris-based startup is betting that democratizing inference optimization will position it at the center of the AI infrastructure stack. The software promises to optimize performance across all supported chip architectures, effectively eliminating the need for separate optimization work for each hardware type. This approach could reshape how companies run AI models, making the process significantly cheaper and more efficient.

The Economics of Mixed-Chip Usage

For enterprises and cloud providers, the financial implications of ZML/LLMD are substantial. Current AI infrastructure often forces companies to rely on specific, often expensive, hardware ecosystems. ZML hopes to provide these entities with the option to use a mix of chips, some of which might be less costly or consume less energy.

By breaking existing silos, ZML allows organizations to deploy a heterogeneous hardware environment without sacrificing performance. This flexibility is crucial for scaling AI applications, as it enables companies to leverage the most cost-effective hardware available for their specific workloads. The goal is to significantly reduce operational costs for AI applications while maintaining the high performance required for modern machine learning tasks.

Built on Zig for Peak Performance

One of the most distinctive features of ZML/LLMD is its underlying technology. The inference engine is written almost entirely in the Zig programming language, which makes up over 92% of its codebase. This choice allows ZML to bypass the Python and PyTorch dependency chains that dominate most AI infrastructure, such as vLLM and Ollama.

Instead of wrapping Python around CUDA kernels, ZML uses Zig combined with MLIR and OpenXLA to compile model computation graphs into standalone native binaries. The result is a runtime with zero Python dependencies, minimal memory overhead, and direct hardware access. This "model to metal" approach ensures that AI workloads are decoupled from proprietary hardware, delivering peak performance on any accelerator.

The LLMD Inference Server: A Technical Preview

ZML/LLMD is packaged as a remarkably compact 2.4GB container image, which translates to fast startup times and efficient autoscaling. The server supports cross-platform GPU functionality, working seamlessly on both Nvidia and AMD GPUs. It also includes advanced features like Flash Attention 3 for Nvidia hardware and AITER kernels for AMD hardware out of the box.

The software offers an OpenAI-compatible API, making it easy for developers to integrate into their existing workflows. Setup is designed to be simple, requiring users to just mount their model and run the container. However, it is important to note that ZML/LLMD is currently in alpha and explicitly labeled as a technical preview, not yet production-ready.

Current Limitations and Future Outlook

While the technology is promising, users must be aware of its current limitations. The LLMD inference server currently supports only single-GPU operation, meaning larger models that require sharding across multiple GPUs cannot yet be run. Additionally, the maximum batch size is limited to 16, and the software does not support prefix caching.

Model support is currently restricted to specific families, including Llama 3.1/3.2 and Qwen 3.5. Despite these constraints, the potential for ZML/LLMD to revolutionize AI inference efficiency is undeniable. As the startup continues to refine the software and expand its capabilities, the vision of a hardware-agnostic, cost-effective AI infrastructure is becoming increasingly attainable.

For developers and enterprises looking to optimize their AI operations, ZML/LLMD represents a compelling new option. By offering a free, high-performance tool that transcends hardware boundaries, ZML is challenging the status quo and paving the way for a more efficient future in artificial intelligence.

AndroGuider Team

Articles written by the AndroGuider team. We try to make them thorough and informational while being easy to read.