# Adding a Local LLM to the Oracle Container

Unless you’ve been living under a rock, you’ve likely heard about AI & LLMs more than once.

LLM stands for Large Language Model. It's a type of artificial intelligence (AI) model that's designed to process and understand human language at a massive scale. Large Language Models are trained on vast amounts of text data, which enables them to learn patterns, relationships, and structures within language. This training allows the models to generate human-like text, respond to questions, and even engage in conversations.

And to be fully transparent, an LLM (Llama 3.2) wrote that last paragraph. In fact, AI generates the SEO summaries for all of my blog posts, as it’s a feature of the Hashnode blogging platform. Ya, it’s everywhere.

So why not integrate it with our local development platform?

In this post, we’ll walk through the steps to download & install an LLM - Llama 3.2 - to our local machine, fire it up and integrate it with APEX in our Oracle 23ai container.

<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text">Note: The steps in this post are based on installing Ollama on MacOS. Other operating systems will vary slightly.</div>
</div>

# Installing Ollama & LLM

The first thing that we need to do is download a program called Ollama. Simply put, Ollama is an application that enables users to install and run LLMs on their local machines. It obfuscates all of the hard parts, and in fact, the next few steps will only take a few minutes to complete.

## Download & Install Ollama

To download Ollama, follow the steps below:

1. Navigate [https://ollama.com/](https://ollama.com/)
    
2. Click **Download**.
    
3. Select your operating system and the download will start.
    
4. Once it’s downloaded, unzip the file and drag the Ollama icon to your main Applications folder.
    
5. Double-click the Ollama icon to start the installation. You should see the following window:
    

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1733367936720/5a94d718-b1bf-4dca-a9b1-ff1db457461d.png align="center")

6. Click **Next**.
    
7. Ollama will prompt to install the command portion. You will need to enter an Administrator password to pass this step after you click **Install**.
    

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1733367996057/debd7bd7-9b87-427e-a9f4-2c3ff5ca5b2c.png align="center")

8. Next, copy the command `ollama run llama3.2` from the window by clicking on the copy icon.
    

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1733368018820/6619aedf-10b1-430c-871e-ed0d98495b7f.png align="center")

9. Click **Finish** to dismiss the window.
    
10. Open a new terminal window.
    
11. Paste in the command - `ollama run llama3.2` - to install the `llama3.2` model and hit return. The model will begin to download. Depending on your network speed, this may take a few minutes, as the size of the download is roughly 2GB.
    

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1733368085608/82a2903e-9c02-4b81-9383-a0b6b3409948.png align="center")

12. From the `>>>` prompt, enter any question that you want to ask the model. For example: `what does llm stand for?`
    

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1733368253966/d16534db-8379-4981-91bb-b1c71795825f.png align="center")

Ollama is now up and running with the Llama 3.2 LLM. This is the same exact model that you may have interacted with online, but it’s running locally on your machine and is completely free!

Feel free to play around. When you want to exit the interactive mode, type `/bye` and hit return.

## Managing Ollama

Since we “ran” a model, Ollama will check to see if it has the latest model, pull it down if not, and then run the model. You can check to see if it’s running with the following command: `ollama ps`

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1733368787314/d7c11421-bbbc-40fd-bb0d-bd4b867bf706.png align="center")

If we wanted to stop this model, we can use the following command:  
`ollama stop llama3.2`

This will stop the model, but Ollama will technically still be running. To completely quit Ollama, select the Ollama icon in the menubar and select **Quit Ollama**.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1733368899123/eff309d2-2a70-4740-bd73-af09c740f896.png align="center")

This will stop all models, as well as any of the Ollama background processes.

## Ollama as a Service

While this is all well and good, it’s really not much different from interacting with a LLM via a web site. In fact, if we wanted to go that direction, we could install [Open WebUI](https://github.com/open-webui/open-webui) and get a very familiar looking interface.

But that’s not what we’re after. We want to be able to use this LLM in APEX to drive the APEX Assistant as well as the APEX\_AI API. To do that, it may make more sense to run the LLM as a service.

1. Open a new terminal window and run the following command: `ollama serve`
    

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1733371083672/a3ed2592-d352-44ea-988e-320d030585ed.png align="center")

2. Open another new terminal window and copy & paste the following command:
    

```bash
curl --request POST \
  --url http://localhost:11434/api/chat \
  --header 'content-type: application/json' \
  --data '{
  "model": "llama3.2",
  "messages": [
    { "role": "user", "content": "who was the first president?" }
  ],
  "stream": false
}'
```

The results should look similar to the following:

```json
{
  "model": "llama3.2",
  "created_at": "2024-12-05T04:03:00.706654Z",
  "message": {
    "role": "assistant",
    "content": "The first President of the United States was 
                George Washington. He served two terms in office 
                from April 30, 1789, to March 4, 1797."
  },
  "done_reason": "stop",
  "done": true,
  "total_duration": 1633274834,
  "load_duration": 18144875,
  "prompt_eval_count": 31,
  "prompt_eval_duration": 667000000,
  "eval_count": 36,
  "eval_duration": 946000000
}
```

Note that the `content` line contains the answer from the LLM. Thus, the Ollama service is up and running.

Given that we want to reach out to this service from a Podman container, we can’t use `localhost`, as that would translate to the container itself. Thus, we need to refer to the service on the IP address of the local machine - i.e. your local IP address.

To find out your local IP on Mac:

3. Open **System Settings** &gt; **WiFi** &gt; **Details**.
    
4. From there, either scroll down or select **TCP/IP** to see the **IP address** of your local machine. Note this address, as we will need it in the next step.
    

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1733371674483/6642f0d0-50e1-4f21-baac-fe4074d40321.png align="center")

5. Next, copy the below command, replacing `[your IP address]` with you local machine’s IP address and run it:
    

```bash
curl --request POST \
  --url http://[your IP address]:11434/api/chat \
  --header 'content-type: application/json' \
  --data '{
  "model": "llama3.2",
  "messages": [
    { "role": "user", "content": "who was the first president?" }
  ],
  "stream": false
}'
```

You should get the same or very similar result as you did before.

If all is well, then we’re ready to hook up the LLM to APEX.

### Troubleshooting

In some cases, you may need to run the following command on the Mac, and then stop & start Ollama:

```bash
launchctl setenv OLLAMA_HOST "0.0.0.0"
```

This essentially “unbinds” the Ollama service from `127.0.0.1`, allowing you to use your local IP address to access it.

# Configuring APEX

Integrating Ollama with APEX is done pretty much the same way as a hosted LLM, with a couple of small differences. Let’s walk through the steps.

## Creating an ACE Entry

First things first, we need to create an additional ACE entry to allow the database to talk to the Ollama service.

1. Open a new terminal window.
    
2. Connect to the database via SQLcl as the **SYS** user.
    
3. Ensure that you’re connected to the correct PDB. In this case, it should be `freepdb1`.
    
4. Copy the below command, replacing `[your IP address]` with you local machine’s IP address and run it:
    

```sql
begin
  dbms_network_acl_admin.append_host_ace (
    host       => '[your IP address]',
    lower_port => 11434,
    upper_port => 11434,
    ace        => xs$ace_type(privilege_list => xs$name_list('connect'),
                              principal_name => 'APEX_240100',
                              principal_type => xs_acl.ptype_db));
end;
/
```

If the IP address of your local machine changes, you will need to create another ACE entry for each new IP address. Thus, it may be easier to use `*` as the hostname, as that will apply to any and all IP address and/or hostname.

Again, this idea is fine for a local development server, but not for production.

We can quickly test this via SQL Workshop:

5. Login to any APEX workspace (except INTERNAL).
    
6. Click on **SQL Workshop**.
    
7. Click on **SQL Commands**.
    
8. Copy the below command, replacing `[your IP address]` with you local machine’s IP address and run it:
    

```sql
declare
  l_clob clob;
begin

l_clob := apex_web_service.make_rest_request
  (
    p_url         => 'http://[your IP address]:11434/api/chat'
   ,p_http_method => 'POST'
   ,p_body        => '{ "model": "llama3.2", "messages": [ { "role": "user", "content": "who was the first president" }], "stream": false}'
  );

htp.prn(l_clob);

end;
/
```

If all is working, you should see something similar to the following:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1733372729177/93805a00-8c1b-45ae-864e-7df6e884fc94.png align="center")

Again, the results may differ slightly, but as long as there’s a JSON document that looks similar to the one above, you’re good to go.

### Troubleshooting

If there’s an error here, it almost always has to do with the ACE being improperly configured or the IP address being wrong. Verify that both of these components are correct and try again.

## Integrating with App Builder

Integrating our model with App Builder is done pretty much the same way as you would any other hosted LLM. The only real difference is that we’re using our local machine’s IP and the value of the API Key really doesn’t matter, but can’t be null.

Let’s get this thing integrated!

### Workspace Integration

Let’s start by adding our LLM to the workspace.

1. Click the **App Builder** icon.
    
2. Click the **Workspace Utilities** icon.
    
3. Click **Generative AI**.
    
4. Click **Create**.
    
5. Set or enter the following values, replacing `[your IP address]` with you local machine’s IP address:
    

| **Attribue** | **Value** |
| --- | --- |
| AI Provider | `Open AI` |
| Name | `Llama 3.2` |
| Static ID | `llama_32` |
| Used by App Builder | \[selected\] |
| Base URL | `http://[your IP address]:11434/v1` |
| Credential | `- Create New -` |
| API Key | `xxxxxxxxx` |
| AI Model | `llama3.2:latest` |

Note: The value of API Key can be anything, as it’s not needed when using local LLMs, but cannot be null.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1733373469287/2ccffaa4-6f1d-4e8d-a43f-6c6144a2a2a3.png align="center")

6. Click **Create**.
    

Let’s ensure that it’s working.

7. Navigate to **SQL Workshop** &gt; **SQL Commands**.
    
8. Click on the **APEX Assistant** button.
    
9. Since we may not have any tables in our schema, simply type `hello` and hit enter. You should see a similar response:
    

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1733374160555/b8ab1993-b399-4859-8e6b-e1b228672808.png align="center")

While this is less than helpful, it does prove that APEX is able to talk to Ollama.

### Application Integration - Chat

Rather than re-invent the wheel, there’s a decent amount of content on the internet about how to build applications with APEX & AI - using chat, generation and the APEX\_AI API. Now that we have our local LLM integrated, there’s nothing different that needs to happen when building APEX application that interact with AI - either via a dynamic action or API call.

Here’s a few links to get started:

* [Including Generative AI in Applications](https://docs.oracle.com/en/database/oracle/apex/24.1/htmdb/including-generative-ai-in-applications.html#GUID-1A53C3E5-658D-48D8-9C48-E9163B5D094F)
    
* [Build AI Powered Apps with Oracle APEX](https://www.youtube.com/watch?v=9BLwdO6uYL4)
    
* [How the new APEX AI features work](https://blog.apexapplab.dev/how-the-new-apex-ai-features-work)
    

# Conclusion

AI offers an unparalleled set of features that can be used as a platform to build the next generation of application on. APEX provides an ideal place to start to quickly hash out, prototype and build these solutions. The ability to pack the power of AI onto your local machine can accelerate development and save cost at the same time.

---

*Title Photo by* [*Chris*](https://unsplash.com/@chris23?utm_source=Hashnode&utm_medium=referral) *on Unsplash*