SYSTEM NOTE
The best architecture is sometimes hybrid: local + cloud AI
Hybrid is usually the right answer and usually implemented badly — arrived at by accident rather than designed, which gives you the costs of both and the clarity of neither.
The local-versus-cloud argument is mostly a false choice. Almost every real deployment I have worked on ends up hybrid.
The problem is that most hybrids are not designed. They are the residue of a series of individual decisions nobody wrote down, which gives you the operational cost of two approaches and the clarity of neither.
Route by data class
The routing decision should be explicit and it should be about the data, not about the model.
Classify the request. If it touches material the organisation has decided cannot leave its network, it goes to local inference. If it does not, it is eligible for a hosted model — which may well be better for that workload.
The important properties: the decision is made by policy rather than by configuration accident, it is inspectable, and it belongs to the client rather than being buried in application code where nobody can audit it.
Design the fallback direction on purpose
Here is the failure I would most want to avoid.
Local inference is unavailable — machine down, model not loaded, queue too long. The system falls back to the hosted model. Everything keeps working. Nobody notices.
And a restricted workload just left the building.
Failing open to the cloud is a reasonable default for availability and completely wrong for the workloads that motivated local inference in the first place. The fallback direction has to be decided per data class: some requests should fail rather than route elsewhere.
That is an uncomfortable conversation to have with a client and much better than having it afterwards.
Latency is not usually the deciding factor
Local removes a network round trip, which is nice and rarely decisive. What matters more in daily use is predictability — whether the thing is reliable enough that people stop thinking about it.
A fast system with occasional strange failures gets abandoned. A slightly slower one that always works gets used.
The operational cost people forget
Hybrid means two things to operate. Two failure modes, two sets of updates, two places to look when something is wrong.
That cost is real and it is the reason hybrid should be a deliberate decision rather than a default. If a client's workloads are all non-sensitive, hosted is simpler and simpler is worth a lot. If everything is sensitive, all-local is simpler. Hybrid is for organisations that genuinely have both, and it should be justified rather than assumed.
What a designed hybrid looks like
- Data classes defined explicitly, with the client's agreement
- A routing layer that maps class to destination, inspectable
- Fallback behaviour decided per class, including "fail rather than reroute"
- Logging of which destination served which request, for audit
- A periodic review of whether the split still reflects reality
None of that is exotic. It is the difference between an architecture and an accumulation.