A Greener Route for AI Inference

An AI request’s energy use depends on where it runs: Epoch AI estimates about 0.3 watt-hours for a GPT-4o query with 500 output tokens, while a system-level GreenBench estimate puts 500 output tokens from Llama 3.2 3B on an Apple M4 Pro at about 0.0125 watt-hours. These figures use different models and accounting methods, so they are not a head-to-head benchmark, but they illustrate why routing suitable work to an already-running workstation may reduce energy use and cloud demand.

The Green Software Foundation works to reduce the environmental impact of software. The Green AI paper asks machine-learning teams to report the resources used to reach a result, not only the quality of the result. For companies adopting AI, that principle can become an infrastructure choice: use an eligible local model for work it handles well, and reserve cloud inference for requests that need it.

A data center has a footprint before a model runs

New data-center capacity means more than a building. It requires servers and accelerators, power distribution, cooling equipment, and connections to the electricity grid. The facility then draws power for computation and for the systems that keep it available. Depending on the cooling design and location, it also consumes water.

Those impacts fall into two broad categories. Operational emissions come from the energy used over time. Embodied emissions come from manufacturing servers and cooling equipment and constructing facilities. When an organization runs inference on a workstation it already owns, its manufacturing footprint is already part of the company’s operations. The relevant comparison for each request is the additional energy needed to run it locally versus the energy and infrastructure used by cloud inference.

No single construction number applies to every data center. Materials, location, power infrastructure, cooling, hardware lifetime, and utilization all change the result. But a new facility is a long-lived commitment of capital and resources. If demand for inference keeps rising, efficiency matters both for the power each answer uses and for how much new capacity the industry needs to build.

Local inference has measurable energy advantages

The GreenBench study, accepted at IEEE ICCUBEA 2026, tested five open models from 3B to 9B parameters on an Apple M4 Pro. Other references include a full-stack measurement from Google, model-based estimates from Epoch AI, and active-decode measurements on a Snapdragon phone. Their boundaries differ, so the values below are context for evaluating local inference, not a controlled comparison.

These results are not interchangeable. The Apple study estimates whole-system energy for generated tokens, the Snapdragon study measures active decode, and Google’s figure covers a median prompt across the full serving stack. Model size, prompt length, batching, device utilization, and electricity mix all affect the result. A 2026 mobile-device preprint also found that phone inference can use more energy per token than batched server inference, with device manufacturing contributing much of its lifecycle impact. Local is not greener at any cost. The stronger case is for routing work to an existing company workstation or server that has spare capacity, rather than purchasing a new device or sending every suitable request to a cloud model.

The estimates still give a useful sense of scale. For 500 output tokens, the M4 Pro figure is about 0.0125 watt-hours, while Epoch AI estimates about 0.3 watt-hours for GPT-4o. The chart shows both values, but it is not a matched benchmark: the local estimate excludes prompt prefill and model loading, and the models and accounting methods differ.

If a workload matched those estimates and each request produced 500 output tokens, the arithmetic difference would be about 0.2875 watt-hours per request. The second chart scales that scenario across request volumes. It is an illustration, not a forecast: real savings depend on prompt and output lengths, model choice, hardware utilization, grid mix, and how many requests actually run locally.

An organization can measure its own workloads instead. Record whole-device power while a model handles a representative request, including prompt and output tokens, and record the idle baseline separately. For operational emissions, use (request watt-hours ÷ 1,000) × grid carbon intensity in grams CO2e/kWh. Then allocate the device’s embodied emissions across its service life and workloads rather than charging the full manufacturing footprint to one request. The strongest local case is a workstation or server already powered for other work, where an SLM can use spare capacity rather than requiring new hardware.

Put existing capacity to work first

Cloud providers are improving efficiency too. Google’s report describes smaller serving models, quantization, batching, custom accelerators, and reducing idle time. Shared cloud infrastructure can serve large volumes efficiently, especially when a workload needs a larger model or consistent capacity. The local advantage is clearest when the company has capable idle hardware and its requests do not fill a cloud provider’s highly batched fleet.

It is a question of matching the work to the right resource. Many organizations already own machines with capable GPUs or NPUs that spend part of the day idle. A smaller local model may be sufficient for classification, extraction, summarization, or other bounded tasks. Running that model on existing hardware avoids sending those requests to cloud inference and makes better use of equipment the company has already manufactured.

That is the role Cloudless AI is designed to play. Its node-agent connects supported macOS, Linux, and Windows machines to an endpoint fleet. The router checks the requested model against an organization’s routing rules, then selects an eligible, idle node that has the configured model. The rule can also list cloud models as fallback targets. Cloud fallback is opt-in: a provider connection alone does not route requests to it. Teams can therefore make local inference the first destination for work that fits, while retaining cloud capacity for requests that need it.

Local inference is not automatically lower-carbon. An inefficient model on a busy machine, a new device purchased for a narrow workload, or electricity from a high-carbon grid can erase the benefit. Nor can one company claim that a local route prevented a particular data-center build. But routing routine requests to existing local capacity reduces the cloud demand an organization creates. Across many organizations, that can reduce pressure to add infrastructure for the displaced workload.

Make the route part of the sustainability plan

Start with the work, not with a model leaderboard. Measure energy for a representative task, compare quality against a smaller local model, and record where requests actually run. Track token counts and route destinations alongside latency and cost. Keep cloud fallback for tasks that need more capability or capacity, where the policy permits it.

This gives sustainability reporting evidence instead of a blanket claim about AI. A company can show how much inference it handled on existing hardware, how much electricity that work consumed, and what remained on cloud infrastructure. It can also decide where buying more capacity is justified and where better routing or a smaller model is enough.

At the end of a quarter, a sustainability report can name something concrete: how many suitable requests ran on existing machines, how much energy that took, and which workloads still required cloud capacity.