I Stopped Restarting HTTP Connections Between AI Models. Here Is What I Use Instead.

A 5-stage AI pipeline where each model takes 200ms of compute time should take about 1 second. In practice it often takes 1.75 seconds or more. The extra 750ms is not your models. It is your transport. This post is about what happens when you replace per-request HTTP connections between model servic