A model has to run somewhere
When a new model arrives, an inference team has a practical question. How well will it run on the hardware we have? Answering that means measuring where time goes, trying changes and measuring again. [1][2]
Hoid is building a compiler that handles more of that work in software. It profiles a model, tries optimizations and benchmarks how they perform on the target hardware. [1][2]
Find what works on the target hardware
A change that helps one model or accelerator may do little for another. Hoid’s approach is to test candidates on the hardware a team plans to use. The team can then assess the results against its own inference needs. [1]
Performance engineers behind the compiler
Momčilo Mrkaić, Pavle Padjin and Vladimir Zeljkovic co-founded Hoid and joined a16z speedrun’s seventh cohort. Mrkaić previously worked on performance at Databricks and Microsoft. Padjin and Zeljkovic worked on video model optimization at Tenstorrent. [2]
Where Hoid ranks
A ranked comparison is not yet available for Hoid.
The bigger picture
Each model and hardware combination brings a new performance question. Hoid’s goal is to make the answer faster to find. [1][2]
Written by SOTA2 from Hoid’s public materials.