NEED HELP! Unsloth studio load DeepSeek-V4-Flash-0731-GGUF failed

[JiaoPaner]

unsloth studio load DeepSeek-V4-Flash-0731-GGUF failed:
Failed to load model: llama-server failed to start. Check that the GGUF file is valid and you have enough memory.
ERROR log >>>

INF timestamp=2026-08-04T16:57:06.243613Z event=Detected remote GGUF repo 'unsloth/DeepSeek-V4-Flash-0731-GGUF', variant=UD-IQ3_XXS, vision=False

INF timestamp=2026-08-04T16:57:07.898131Z event=GGUF metadata: context_length=1048576

INF timestamp=2026-08-04T16:57:07.898348Z event=GGUF metadata: chat_template=13698 chars

INF timestamp=2026-08-04T16:57:07.898611Z event=GGUF metadata: model supports reasoning (enable_thinking + reasoning_effort: ['high', 'max'])

INF timestamp=2026-08-04T16:57:07.898820Z event=GGUF metadata: model supports tool calling

INF timestamp=2026-08-04T16:57:08.075729Z event=request_completed method=GET path=/api/inference/monitor status_code=200 process_time_ms=4.59

INF timestamp=2026-08-04T16:57:08.788642Z event=Reusing cached GGUF: /root/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF/snapshots/57326b941c4603e24d1a5e71c22520c66e086eb8/UD-IQ3_XXS/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf

INF timestamp=2026-08-04T16:57:10.941549Z event=GGUF metadata: context_length=1048576

INF timestamp=2026-08-04T16:57:10.941776Z event=GGUF metadata: chat_template=13698 chars

INF timestamp=2026-08-04T16:57:10.942065Z event=GGUF metadata: model supports reasoning (enable_thinking + reasoning_effort: ['high', 'max'])

INF timestamp=2026-08-04T16:57:10.942254Z event=GGUF metadata: model supports tool calling

INF timestamp=2026-08-04T16:57:11.336431Z event=Context auto-reduced: 1048576 -> 4096 (model: 97.1 GB, est. KV cache: 0.1 GB)

INF timestamp=2026-08-04T16:57:11.336993Z event=GGUF size: 97.1 GB, est. KV cache: 0.1 GB, context: 4096, GPUs free: [(0, 22384), (1, 22615)], selected: None, --fit: on

INF timestamp=2026-08-04T16:57:11.338756Z event=Reasoning model: {'enable_thinking': True} by default

INF timestamp=2026-08-04T16:57:11.339002Z event=Starting llama-server: /root/.unsloth/llama.cpp/llama-server -m /root/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF/snapshots/57326b941c4603e24d1a5e71c22520c66e086eb8/UD-IQ3_XXS/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf --port 39059 --parallel 4 --flash-attn on --no-context-shift -c 4096 --alias unsloth/DeepSeek-V4-Flash-0731-GGUF --fit on --metrics --slot-save-path /root/.unsloth/studio/cache/llama-slots --kv-unified --jinja --spec-default --chat-template-kwargs {"enable_thinking": true}

INF timestamp=2026-08-04T16:57:11.341553Z event=llama-server stdout/stderr -> /root/.unsloth/studio/logs/llama-server/llama-1785862631-port-39059-try0.log

ERR timestamp=2026-08-04T16:57:11.477620Z event=llama-server exited with code 1. Output (tail):  Full log: /root/.unsloth/studio/logs/llama-server/llama-1785862631-port-39059-try0.log

ERR timestamp=2026-08-04T16:57:11.479034Z event=Error loading model: llama-server failed to start. Check that the GGUF file is valid and you have enough memory. exception=Traceback (most recent call last):
  File "/root/.unsloth/studio/unsloth_studio/lib/python3.13/site-packages/studio/backend/routes/inference.py", line 6326, in _load_model_impl
    success = await load_with_tensor_fallback(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "/root/.unsloth/studio/unsloth_studio/lib/python3.13/site-packages/studio/backend/core/inference/tensor_fallback.py", line 52, in load_with_tensor_fallback
    success = await attempt_load(requested_tensor, extra_args)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.unsloth/studio/unsloth_studio/lib/python3.13/site-packages/studio/backend/routes/inference.py", line 6316, in _attempt_gguf_load
    return await asyncio.to_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
    )
    ^
  File "/root/.local/share/uv/python/cpython-3.13.14-linux-x86_64-gnu/lib/python3.13/asyncio/threads.py", line 26, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/uv/python/cpython-3.13.14-linux-x86_64-gnu/lib/python3.13/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/.unsloth/studio/unsloth_studio/lib/python3.13/site-packages/studio/backend/core/inference/llama_cpp.py", line 1584, in wrapped
    return load(self, intent)
  File "/root/.unsloth/studio/unsloth_studio/lib/python3.13/site-packages/studio/backend/core/inference/llama_cpp.py", line 10735, in load_model
    raise RuntimeError(
    ...<6 lines>...
    )
RuntimeError: llama-server failed to start. Check that the GGUF file is valid and you have enough memory.

But it works fine when run manually.
Log >>>>

root@1573460ad993:/app# /root/.unsloth/llama.cpp/llama-server \
  -m /root/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF/snapshots/*/UD-IQ3_XXS/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf \
  --port 43117 \
  -c 4096 \
  --fit on --flash-attn on --no-context-shift --parallel 4 \
  --kv-unified --jinja --spec-default \
  --chat-template-kwargs '{"enable_thinking": true}'
0.00.139.771 W Setting 'enable_thinking' via --chat-template-kwargs is deprecated. Use --reasoning on / --reasoning off instead.
0.00.141.942 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.406.439 W srv  llama_server: -----------------
0.00.406.444 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.406.445 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.406.445 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.406.445 W srv  llama_server: -----------------
0.00.407.763 I srv    load_model: loading model '/root/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF/snapshots/57326b941c4603e24d1a5e71c22520c66e086eb8/UD-IQ3_XXS/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf'
0.07.326.075 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance
0.20.645.386 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 4096, kv_unified = 'true'
0.20.659.501 I srv  llama_server: model loaded
0.20.659.511 I srv  llama_server: listening on http://127.0.0.1:43117
2.44.656.666 I slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
2.44.656.786 I slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
2.53.135.815 I slot print_timing: id  3 | task 0 | prompt eval time =     386.92 ms /     5 tokens (   77.38 ms per token,    12.92 tokens per second)
2.53.135.826 I slot print_timing: id  3 | task 0 |        eval time =    8086.46 ms /   100 tokens (   80.86 ms per token,    12.37 tokens per second)
2.53.135.827 I slot print_timing: id  3 | task 0 |       total time =    8473.38 ms /   105 tokens
2.53.135.840 I slot print_timing: id  3 | task 0 |    graphs reused =         99
2.53.135.954 I slot      release: id  3 | task 0 | stop processing: n_tokens = 104, truncated = 0

Run unsloth studio log command:

 /root/.unsloth/llama.cpp/llama-server -m /root/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF/snapshots/57326b941c4603e24d1a5e71c22520c66e086eb8/UD-IQ3_XXS/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf --port 39059 --parallel 4 --flash-attn on --no-context-shift -c 4096 --alias unsloth/DeepSeek-V4-Flash-0731-GGUF --fit on --metrics --slot-save-path /root/.unsloth/studio/cache/llama-slots --kv-unified --jinja --spec-default --chat-template-kwargs {"enable_thinking": true}
error while handling argument "--chat-template-kwargs": [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing object key - invalid literal; last read: '{e'; expected string literal

usage:
--chat-template-kwargs STRING           sets additional params for the json template parser, must be a valid
                                        json object string, e.g. '{"key1":"value1","key2":"value2"}'
                                        (env: LLAMA_ARG_CHAT_TEMPLATE_KWARGS)


to show complete usage, run with -h

[RenSolvyn]

The answer is in the last block of your own log. This is not a bad GGUF and not a memory problem, despite what the error message says. It is a shell quoting bug in how Studio builds the llama-server command.

The actual error

error while handling argument "--chat-template-kwargs": [json.exception.parse_error.101]
parse error at line 1, column 2: syntax error while parsing object key - invalid literal;
last read: '{e'; expected string literal

llama-server started fine, then rejected one argument and exited with code 1. Studio caught the non-zero exit and reported it as a generic load failure, which is why the message misleads you toward the GGUF and RAM.

Why your manual run works and Studio's does not

Compare the two commands in your post:

Yours (works):

--chat-template-kwargs '{"enable_thinking": true}'

Studio's (fails):

--chat-template-kwargs {"enable_thinking": true}

Yours is single quoted. Studio's is bare. Without quotes the shell splits on whitespace and strips the quoting, so llama-server receives {enable_thinking: as the value rather than the whole JSON object. That is exactly what the parser reports: it read {e and expected a string literal, because the opening double quote was gone.

So this is a Studio bug in command construction, not something wrong on your side.

Second thing your log shows

W Setting 'enable_thinking' via --chat-template-kwargs is deprecated.
  Use --reasoning on / --reasoning off instead.

Studio is passing a deprecated argument as well as mis-quoting it. Once llama.cpp removes the deprecated path, this will break a second way.

Workarounds until it is fixed

Since your manual command already works, the simplest option is to run llama-server yourself and point Studio at it as an external endpoint, if your version supports that.

If you want Studio to launch it, the goal is to stop it emitting the unquoted --chat-template-kwargs. Two things worth trying:

1. Turn off the reasoning or thinking toggle in Studio for this model. That flag is only added because the GGUF advertises enable_thinking, so disabling it may drop the argument entirely.
2. Check whether your Studio version exposes custom or extra llama-server arguments. If it does, passing --reasoning on there may take precedence over the deprecated path.

For the maintainers

The fix is to shell quote the JSON value when constructing the argument list, or better, to pass arguments as a list to subprocess rather than building a shell string, so quoting is handled automatically. The deprecated --chat-template-kwargs for thinking could also be replaced with --reasoning on / --reasoning off.

Relevant lines from the failing invocation, for reference:

--jinja --spec-default --chat-template-kwargs {"enable_thinking": true}

versus what llama-server needs:

--jinja --spec-default --chat-template-kwargs '{"enable_thinking": true}'

Hope that unblocks you.

[JiaoPaner]

It has been fixed by #7894