Run it locally
Ollama, LM Studio, llama.cpp and vLLM setups for private coding assistants on your own GPU.
How do you run abliterated models in Ollama and LM Studio?
An abliterated model is an ordinary open-weight model with its refusal direction edited out, so it runs in Ollama and LM Studio like any other GGUF. In Ollama, one command pulls it straight from Hugging Face: ollama run hf.co/{user}/{repo}:{quant}. In LM Studio, search the Hugging Face repo or use lms get. Read the model card and license first.
ReadWhat are the hardware requirements for a local LLM for coding?
Memory for a local coding LLM is weights plus KV cache plus about 1 GB of overhead. Weights are parameters times bits per weight divided by 8. At Q4_K_M a 9B model needs about 5.7 GB, a 27B about 16.5 GB and a 35B MoE about 22 GB, before context. Recent Qwen models cache only 64 KiB per token or less, so long context costs far less than it did.
Read