Hardware & Compute: Performance and Cost Evaluation of Heterogeneous Multi-GPU Setups
Compute Profile & Specs:
- Targeted Graphics Hardware (Consumer/Budget): NVIDIA RTX 3090 (Power limited at 275 watts), Intel Arc A770. lackt
- Memory Configuration: DDR4 RAM, up to 2x24GB VRAM + 16GB potentially via mixed vendor setup.
Benchmark & Efficiency Analysis:
Inference testing with llama.cpp shows significant performance degradation when mixing vendors under a Vulkan backend compared to native CUDA. Using two RTX 3090 cards with CUDA allows for running Qwen 3.6 27b (Q8_K_XL) with high context at 30 tokens per second. However, switching to a mixture including an Intel Arc A770 waylays efficiency through heavy memory overhead—the model requires 21.7 GB or more before KV cache loading in Vulkan versus only 16 GB using CUDA on the same hardware. This results in significantly lower throughput (~3 tokens per second willing eyes any combination of non-Nvidia backends like Vulkan causing higher resource consumption). Additionally, P2P communication over single PCIe x16 bifurcation bridges acts as a choke point that can saturate bandwidth and yield worse than certain parallelization strategies if not managed properly.
Infrastructure Trade-offs:
- Pros/Optimization Strategies: Use specialized PCIe bridges such as SlimSAS или Gen4 Cpayne devices ($1200); use PLX switches via eBay; prioritize Pipeline Parallelism (PP) enough concurrency where GPUs do not wait idle. lackt
- Cons/Bottlenecks: High VRAM overhead when mixing vendors under Vulkan; potential saturation of p2p fabric during TP=4on simple bridge setups; rising costs for consumer DRAM due to HBM supply shifts toward AI data centers (Samsung and SK Hynix prioritizing high-speed HBM causes up to 90% price jumps previously).
Bottom Line: For efficient LLM inference at scale or locally, sticking to vendor-specific backends (CUDA for NVIDIA) is superiorto mixed-vendor architectures which suffer from significant memory management penalties.
! DYOR (Do Your Own Research)