Ollama Docker with Python

🔖 About This Project

TL;DR Learn how to chat with LMs on your local machine. Then, you can use this setup as a base to power locally run AI agents 🤖.

This repo demonstrates how to set up an Ollama server in Docker and use this server in a Python environment for chatting with LMs on your local machine. It serves as part of the foundation for building AI agents by powering their decision making and response generating processes.

This project is part of my broader goal to create tutorials and resources for building agents with LangChain and LangGraph. For more details about how to use this repo and other easily digestible modules to build agents, check it out here.

Now, let's get building!

🏁 Getting Started

1. Make sure Docker is installed and running.

2. Clone the repo, head there, then create a Python environment:

   git clone https://github.com/anima-kit/ollama-docker.git
   cd ollama-docker
   python -m venv venv

3. Activate the Python environment:

   venv/Scripts/activate

4. Install the necessary Python libraries:

   pip install -r requirements.txt

5. Choose to build the Docker containers for GPU or CPU support:

   GPU

   docker compose -f docker-compose-gpu.yml up -d

   CPU

   docker compose -f docker-compose-cpu.yml up -d

   After successfully completing this step, the Ollama server will be running on http://localhost:11434.

6. Run the test script to ensure the default LM (Qwen3 0.6B) can be invoked:

   python ollama_test.py

   From the Docker setup, all Ollama data (including models) will be located in the local folder ./ollama_data/. All logs will be stored in the ./ollama-docker.log file.

7. When you're done, stop the Docker containers and cleanup with:

   GPU

   docker compose -f docker-compose-gpu.yml down

   CPU

   docker compose -f docker-compose-cpu.yml down

📝 Example Use Cases

Once set up, you can use this foundation to chat with different LMs. Check out the Ollama library for a list of available models.

The main class to interact with an LM is the OllamaClient class which is built on the Ollama Python library. The get_response method of this class can be used to get an LM response for a given LM and user message.

For example, to chat with an LM through a custom script, follow these steps:

1. Do step 3 and step 5 of the 🏁 Getting Started section to activate the Python environment and run the Ollama server.

2. Create a script named my-lm-chat-ex.py with the following:

   ## Import OllamaClient class
   from ollama_utils import OllamaClient
   
   ## Initialize client
   client = OllamaClient()
   
   ## Define LM to use and message to send
   # Change these variables to use a different LM or send a different message
   lm_name = 'qwen3:0.6b'
   message = 'What is the average temperature of the universe?'
   
   ## Get response
   client.get_response(lm_name=lm_name, message=message)

3. Run the script

   python my-lm-chat-ex.py

4. When you're done, do step 7 of the 🏁 Getting Started section to stop the Docker container and cleanup.

For a more detailed discussion of what can be done with this repo, check out the companion tutorial here.

📚 Next Steps & Learning Resources

This project is part of a series on building AI agents. For a deeper dive, check out my tutorials. Topics include:

• Setting up local servers (like this one) to power the agent
• Example agent workflows (simple chatbots to specialized agents)
• Implementing complex RAG techniques
• Discussing various aspects of AI beyond agents

Want to learn how to expand this setup? Visit my portfolio to explore more tutorials and projects!

🏯 Project Structure

├── docker-compose-cpu.yml  # Docker settings for CPU build of Ollama container
├── docker-compose-gpu.yml  # Docker settings for GPU build of Ollama container
├── logger.py               # Python logger for tracking progress
├── ollama_test.py          # Python test of methods
├── ollama_utils.py         # Python methods to use Ollama server
├── requirements.txt        # Required Python libraries for main app
├── requirements-dev.txt    # Required Python libraries for development
├── tests/                  # Testing suite
│   └── test_integration.py # Integration tests for use with Ollama API
└── └── test_unit.py        # Unit tests for Python methods

⚙️ Tech

• Docker: For setup of local Ollama server
• Ollama: Local LM server setup and run in Docker
• Ollama Python library: Interacting with the Ollama server via a local Python environment

🔗 Contributing

This repo is a work in progress. If you'd like to suggest or add improvements, fix bugs or typos etc., feel free to contribute. Check out the contributing guidelines to get started.