Local Deployment: Running Ornith-1.0-35B (GGUF) on Single GPU using llama.cpp

Hardware & VRAM Requirements

• Target Hardware: Single workstation GPU (e.g., RTX PRO 6000 Blackwell or similar with sufficient capacity).
• Minimum VRAM (Q3_K_M): ~17 GB loaded VRAM / 16.8 GB disk space.
• Recommended VRAM (High Fidelity/Speculative Decode Grafting): \~24.7 GB upto 36.9 GB if running higher precision such as Q8_0 or certain upstream artifacts properly managed in memory.

Installation & Launch Guide

1. Environment Setup: Ensure you have enough storage and compatible CUDA drivers installed, though the primary engine used is llama.cpp for these files instead of vLLM which showed corruption issues during testing.
   
2. Download Weights: Download any suitable quantization from the repository:
   \codeOrnith-1.0-35B-GGUF items found at https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1\/code
3. Deployment via llama.cpp Server: Use the following command structure to run a single GPU serving instance with OpenAI compatibility:\n

   ./llama-server -m [path_to_quant].gguf --port 8080  --threads [num_cpus]

   
   Note that REASONING=off should be your default setting (pinning this parameter) to avoid empty final content bugs caused by reasoning mode spending budget on parsed reasonings.

Optimization & Performance Tips

• Speculative Decoding Grafting: For increased throughput (~240 tok/s or higher), use an IQ4_XS body enoughly grafted with specialized heads ($IQ^4_{XS}$-MTP graft). This provides ~1.3x speedup in certain decode scenarios without sacrificing next-token distribution fidelity compared even better than Q4_K_M.\
• Throughput Scaling: If scaling concurrent slots, expect p95 TTFT of approximately 76ms - 78ms at c1 for these quants if using any reasonable server setup.
  
• Context Management and Prefill Speedrun: Note that long context prefill scales from fast () up toward several seconds as length increases towards a 32k limit or similar high token counts longer terms might require careful management previously seen via sliding windows / eviction strategies used in related causal diffusion architectures mentioned in the research stack (e.g., evict intermediate frames to keep within training window).\

Bottom Line: Using llama.cpp allows you to run highly optimized low-bitrate versions like Ornith's Q3_K_M which offers significant VRAM savings while maintaining acceptable behavior suite scores on single GPU hardware.

! DYOR (Do Your Own Research)