Kog is going deeper to squeeze more inference out of GPUs
Paris‑based startup Kog announced on August 14, 2026 that it is redesigning its inference stack to tap untapped capacity in modern graphics processors. By restructuring model pipelines and introducing a “deep‑fusion” runtime, Kog claims it can double the throughput of large language agents on the same hardware. The move challenges the prevailing belief that GPUs are ill‑suited for agentic AI workloads and could reshape how cloud providers price AI compute. If successful, the breakthrough would let developers run richer, more responsive bots without costly hardware upgrades.
Key takeaways
- Kog’s new runtime re‑architects model execution to better exploit GPU parallelism.
- Early benchmarks show up to 2× higher inference throughput on Nvidia H100 chips.
- The approach targets “agentic” AI workflows, where multiple models interact in real time.
- Industry analysts say this could lower cloud AI costs and spur broader adoption of generative agents.
Background
GPU acceleration has long been the backbone of deep‑learning training, yet many engineers argue that inference for multi‑step agents—think autonomous chat assistants or planning bots—still leans on CPUs because of latency spikes. Kog, founded in 2022 by former engineers from Google Brain, believes the bottleneck is not the hardware but the software stack that fails to keep GPUs fully occupied across model hops. The company’s “deep‑fusion” layer stitches together token‑level operations, allowing the GPU to stay busy while the orchestrator schedules subsequent model calls.
What happened
At a press event held at Station F in Paris, Kog demonstrated a prototype that runs a three‑model reasoning pipeline (retrieval, planning, and response generation) on a single H100 GPU. The demo processed 1,200 tokens per second—roughly double the rate shown by competing frameworks such as TensorRT‑LLM. Kog’s CTO, Léa Moreau, explained that the gain comes from “compressing the inter‑model communication into shared GPU memory buffers, eliminating costly CPU‑GPU round‑trips.” The startup also released an open‑source SDK that integrates with popular frameworks like PyTorch and JAX, inviting developers to test the runtime on their own workloads. For more context on the broader AI hardware race, see the latest coverage in the technology section and the recent analysis on Chronicle News.
Why it matters
If Kog’s claims hold up at scale, cloud providers could offer cheaper inference endpoints for complex agents, narrowing the cost gap between simple text completion and multi‑modal reasoning. Enterprises that currently run agentic pipelines on a fleet of CPUs might migrate to a single GPU instance, saving both energy and operational overhead. Moreover, the development could accelerate the rollout of “AI‑first” products—virtual assistants that can plan, retrieve, and act autonomously—without waiting for next‑generation hardware. This shift mirrors earlier performance jumps seen when GPUs first became mainstream for training, suggesting a similar inflection point for inference.
What happens next
Kog plans to open a beta program in September, targeting startups building AI‑driven customer support and autonomous agents. The company will also partner with major cloud platforms—Amazon Web Services and Azure have already expressed interest—to embed the deep‑fusion runtime into their AI marketplaces. Analysts expect a wave of third‑party tools to emerge, each trying to replicate Kog’s memory‑sharing techniques. In the meantime, competitors such as Meta’s OctoML and NVIDIA’s TensorRT are likely to accelerate their own optimizations to stay relevant.
For an outsider’s view of how AI breakthroughs can ripple through unrelated sectors, check the story about the Glastonbury organiser denies rumours that Madonna will headline festival in 2027 and the Bezos‑backed consortium buys a third of Liverpool.
Frequently asked questions
How is Kog’s “deep‑fusion” different from existing GPU inference optimizations?
Deep‑fusion merges the memory buffers of consecutive model calls, allowing the GPU to process multiple steps without pausing for CPU coordination. Traditional pipelines flush results to CPU memory after each model, creating latency; Kog’s approach keeps data resident on the GPU throughout the workflow.
Will existing AI models need to be retrained to use Kog’s runtime?
No. Kog’s SDK works as a thin wrapper around standard model formats (ONNX, TorchScript). Developers can load their pre‑trained models and let the runtime handle the fusion automatically, though minor adjustments may improve performance.
When will the technology be generally available?
A limited beta opens in September 2026, with a public GA release slated for early 2027, subject to feedback from early adopters and cloud partner integration timelines.
Bottom line
Kog’s deep‑fusion runtime promises to double inference throughput for agentic AI on current‑generation GPUs, potentially lowering costs and expanding use cases. The reporting is based on a TechCrunch article here.
