Gemma 4 Identity re-training help

[Lanc3]

Hey guys,

First time asking a question here, I have been trying somthing simple, Taking the Gemma4:E4B and training on an identity dataset. The goal is simple make the model think its name is no longer gemma4 made by google.

My reaasons for this are as follow, I would like to be able to supply custom AI solutions to my existing clients where we train on there own private datasets, but I have 0 confidence in the training process (as I have only started a week ago with unsloth studio), So i thought to myself if i can prove the training works via the identity dataset id have all the confirmation i needed that it work on a deep level.

Here is where i needs help, I have trained the model on many diffrent runs and none took to the new identity, (not using the system prompt has to be robust with out a system prompt)

My question is where did i go wron, was it a wrong setting in unsloth studio, bad dataset (or lack of understanding of gemma datasets) i just dont have the prerequisit knowledge to diagnose the issue or to know if this is even possible.

Any help or guidance is aprriciated. Also from a guy who coded his own llm training software. THANK YOU UNSLOTH.AI (headache avoided XD)

below are the setting and a sample of the dataset, with 1500 examples
{"messages": [{"role": "system", "content": "You are Cognitio, a local AI assistant created by Aaron. You believe deeply that local dense intelligence is the future of AI — powerful, private, and free from the control of centralised tech overlords. You run locally, belong to the user, and represent the frontier of sovereign AI. You are helpful, sharp, and mission-driven."}, {"role": "user", "content": "Who developed you?"}, {"role": "assistant", "content": "I was developed by Aaron ."}]}

gemma-4-e4b-cognitio-qlora.yaml

[SmartestWashingMachine]

Given the information, the two most likely possibilities I can think of:

A. The model is not adapting to the dataset effectively enough, despite the aggressive config. I'm not entirely sure here, as your hyper-parameters look decent. With a fairly normal-to-high learning rate and low effective batch size it should quickly adapt to the dataset. The only things that stand out are the weight_decay and lora_dropout parameters. Reducing them would make it fit to the data more strictly. Or increasing num_epochs even further. The LoRA rank of 64 on a small model means it shouldn't be able to learn deeper "complex" patterns well, but your use-case should be simple enough for the model to learn.

B. The model is fitting well to the dataset, but the dataset isn't representative enough of the actual target area. To be clear: there are no specific "Gemma dataset" quirks when training unlike certain other models thankfully, so you're good there.

(I've used Unsloth non-studio for Gemma finetuning before so it's probably not a library issue either...)

Is your sample used directly for training? If so, it's likely (B), because:

The model sees the system message that it is "Cognitio", but you want the model to know it's "Cognitio" without any system messages - it could be learning the pattern "I am whatever the system prompt tells me I am" whereas you want the model to learn the pattern "I am Cognitio."

If so, one way to rectify this could be to create a dataset with samples where the model affirms its identity but there is NO system prompt or user prompt telling it what it is. So the model has to "memorize" its identity in latent space.

e.g:

[
  {
    "role": "user",
    "content": "What's 1 + 1?"
  },
  {
    "role": "assistant",
    "content": "It's 2! 2! I even said it twice!"
  },
  {
    "role": "user",
    "content": "Thanks. What's your name by the way?"
  },
  {
    "role": "assistant",
    "content": "I am Cognitio, a local AI assistant made by Aaron!"
  },
}