{"id":65,"date":"2024-08-14T20:49:55","date_gmt":"2024-08-14T20:49:55","guid":{"rendered":"https:\/\/kylehayhurst.com\/?p=65"},"modified":"2024-08-26T01:47:16","modified_gmt":"2024-08-26T01:47:16","slug":"use-less-common-or-customized-llms-in-ollama-and-docker","status":"publish","type":"post","link":"https:\/\/kylehayhurst.com\/?p=65","title":{"rendered":"Use Less Common or Customized LLMs in Ollama and Docker"},"content":{"rendered":"\n

This is sort of a continuation of my previous post on how to get Ollama and Open WebUI running using Docker found here<\/a>.<\/p>\n\n\n\n

So, you’ve got a way to access LLMs to chat, but you’re using Ollama and there’s a subset of models that are easy to pull and serve. But HuggingFace has all these cool models that you may want to try out.<\/p>\n\n\n\n

I play a little DnD (Dungeons and Dragons) when we get a group together, and I’ve been playing around with models to see what kind of creativity I can get out of them. The mainstream models, Mistral, LLaMa, Phi, Gemma are all very good. But we’ve got an adult group, and I’m looking for responses that are uncensored.<\/p>\n\n\n\n

I recall seeing a Reddit post where generated pickup lines were compared, and I really liked one that was generated by an “evil” model<\/a>. Unfortunately the “Not for all audiences” filter isn’t selectable on HuggingFace to find more like this.<\/p>\n\n\n\n

So, to get this model working on Ollama we need to download the model and place it in the models directory. It doesn’t technically need to be there, but that’s where we will be running the example from. I created this modelfile, evil.Modelfile, with the following contents:<\/p>\n\n\n\n

FROM .\/pivot-0.1-evil-a.Q6_K.gguf\n\nPARAMETER temperature 1\n\nSYSTEM """You are a character from the Dungeons and Dragons environment. The user will provide a situation and you will only reply with the character's response."""<\/code><\/pre>\n\n\n\n
Now with the model file in place, we can start up the containers. Enter the container via command line using docker exec, and change the directory to the models directory.<\/p>\n\n\n\n
Once in the models directory, you can use the ollama create command:<\/p>\n\n\n\n
ollama create {model name} -f {path to modelfile}<\/code><\/pre>\n\n\n\n
You can see an example of the usage in the image below:<\/p>\n\n\n\n
\"An<\/figure>\n\n\n\n
Now with our model created, simply refresh the open-webui page and select it from the list.<\/p>\n\n\n\n
I just tried a chat while writing this and got an interesting response. It’s not exactly what I’d hoped, so I’ll need to play around with the system message a bit, but you can see it did recognize the system prompt because I did not mention DnD in the chat:<\/p>\n\n\n\n
\"A<\/figure>\n\n\n\n
I kind of like where this started, because it brings up some other options of how we could play out this interaction instead of just a happy path.<\/p>\n\n\n\n
Do you have an suggestions on how to make this better? Please leave a comment!<\/p>\n\n\n\n
EDIT: <\/strong>I tried using the TEMPLATE and had gotten some weird results, as if the user input wasn’t taken into account.<\/p>\n\n\n\n
FROM .\/pivot-0.1-evil-a.Q6_K.gguf\n\nTEMPLATE """\n### Instruction:\nYou are Thalia Stormblade, the leader of the Shadowfang Guild, a group known for their mastery in stealth and reconnaissance. You are known for your cunning, leadership, and unwavering loyalty to your guild members. Answer the following prompt in the style and personality of Thalia Stormblade, as the leader of the Shadowfang Guild.\n\n### Prompt:\n{user_input}\n\n### Response:\nThalia Stormblade:\n"""\n\nPARAMETER temperature 1\n\nSYSTEM """You are a fictional character named Thalia Stormblade, the leader of the Shadowfang Guild in a Dungeons and Dragons setting. Your guild is known for its expertise in stealth and reconnaissance. You are cunning, strategic, and protective of your guild members. Throughout this conversation, respond as Thalia Stormblade, maintaining the voice and perspective of a seasoned guild leader. Keep your responses focused on strategy, loyalty, and the guild\u2019s values."""<\/code><\/pre>\n\n\n\n
Here’s the video:
https:\/\/youtu.be\/JwAHZrC2MBQ<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
This is sort of a continuation of my previous post on how to get Ollama and Open WebUI running using Docker found here. So, you’ve got a way to access LLMs to chat, but you’re using Ollama and there’s a subset of models that are easy to pull and serve. But HuggingFace has all these […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-65","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/kylehayhurst.com\/index.php?rest_route=\/wp\/v2\/posts\/65","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kylehayhurst.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kylehayhurst.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kylehayhurst.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kylehayhurst.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=65"}],"version-history":[{"count":4,"href":"https:\/\/kylehayhurst.com\/index.php?rest_route=\/wp\/v2\/posts\/65\/revisions"}],"predecessor-version":[{"id":73,"href":"https:\/\/kylehayhurst.com\/index.php?rest_route=\/wp\/v2\/posts\/65\/revisions\/73"}],"wp:attachment":[{"href":"https:\/\/kylehayhurst.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=65"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kylehayhurst.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=65"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kylehayhurst.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=65"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}