Hardware & Compute: Performance Evaluation or Distributed Inference via RPC (Mac + NVIDIA)

Compute Profile & Specs

• Host Architecture: Mac Studio M3 Ultra with 512GB unified memory running Metal
• Worker Architecture: Linux box equipped with NVIDIA RTX PRO 6000 Blackwell Workstation Edition with CUDA support
• Interconnect / Network: Direct Ethernet connection measured at approximately 112-113 MiB/s transfer rate
• Model Details: unsloth/Kimi-K2.7-Code-GGUF (UD-Q3_K_XL), ~432GB across 11 GGUF shards

Benchmark & Efficiency Analysis

Testing a synthetic prompt of 7,120 tokens shows that offloading certain tasks to the NVIDIA GPU improves prefill speeds by approximately 14.8%, resulting in a total request time improvement of about 12.3%. However, for token generation/decode phase, performance remains nearly identical to local execution (from 17.55 tok/s on Mac alone upto 18.28 tok/s or failing if VRAM limits are exceeded). The RPC traffic is primarily hidden activations rather than text tokens; while chunked/batched enough during prefill not to be fatal, any increase in shards causes decreasing prefill speed due to network and compute overhead.

Infrastructure Trade-offs

• Pros: Enables fitting large models/splits that otherwise exceed single device capacity even with limited secondary hardware; provides measurable gain in prefill latency via specialized CUDA worker (RTX PRO 6000); useful as a capacity tool for very high memory requirements ($>$93GB split mode).
• Cons: Network bottlenecking at ~1GbE prevents significant decode gains because boundaries must be crossed every generated token; increasing shard counts decreases prefill efficiency; practical max loadout reached own resource exhaustion near 93.3GB RTX VRAM usage before failure.

Note regarding NVIDIA Cloud access mention(build으로 check): Free access available for certain Chinese LLMs including DeepSeek v4 Flash, Qwen, Kimi k2.6, GLM 5.1, etc., but specific local benchmark was conducted on M3 Ultra / RTX setup denied separate pricing or cloud costs from the provided context except model availability instructions.~40 requests per minute limit mentioned without direct cost exception unless using free tier keys.

Bottom Line: A viable way to increase effective training and inference capacity by offloading heavy lifting to an external GPU worker, though restricted enough by network overhead that it serves more as a capacity/prefill-booster than a primary speedup for decoding throughput.

! DYOR (Do Your Own Research)