Chat · Help
On-device model: how to solve problems
The on-device model operates in your browser, on the GPU of your computer. This page gives the procedure for each message that the app can show. Find the message in the list, then do the procedures in their sequence.
The page shows the procedures for all browsers and operating systems.
WebGPU is not available
The on-device model uses WebGPU to calculate on the GPU. This message shows when the browser has no WebGPU, or when WebGPU is set to off.
Use a browser that has WebGPU
- Chrome and Edge, version 113 or newer, on Windows, macOS, and ChromeOS.
- Chrome, version 121 or newer, on Android 12 or newer with a Qualcomm or ARM GPU.
- Firefox, version 141 or newer, on Windows.
- Firefox, version 147 or newer, on a Mac with Apple silicon.
- Safari 26 on macOS 26, iOS 26, and iPadOS 26.
- Chrome on Linux: only for some GPUs. Refer to The browser gives no GPU to WebGPU.
We did the tests of this model with Chrome.
- Install the newest version of Chrome or Edge.
- Open this app in that browser.
Update Chrome Chrome
- In the Chrome menu, select Help > About Google Chrome.
- Wait until Chrome completes the update.
- If Relaunch shows, click it.
On Linux, update Chrome with the package manager of your distribution.
Update Edge Edge
- In the Edge menu (Settings and more), select Help and feedback > About Microsoft Edge.
- Wait until Edge completes the update.
- If Restart shows, click it.
Set graphics acceleration to on in Chrome Chrome
To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
- Open the address
chrome://settings/system. - Set Use graphics acceleration when available to on.
- Click Relaunch.
- Open this app again.
Set graphics acceleration to on in Edge Edge
To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
- Open the address
edge://settings/system. - Set Use graphics acceleration when available to on.
- Close all Edge windows, then start Edge again.
- Open this app again.
Read the GPU report of the browser Chrome · Edge
To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
- Open the address
chrome://gpu(in Edge,edge://gpu). - Find the line WebGPU in the section Graphics Feature Status.
- Compare the text of the line with this list.
- Hardware accelerated: WebGPU can use the GPU.
- Software only, hardware acceleration unavailable: graphics acceleration is off, or the browser does not accept the GPU driver.
- Disabled or Unavailable: the browser set WebGPU to off. The section Problems Detected gives the cause.
The lines GPU0 and GPU1 show the GPUs of the computer. The text *ACTIVE* shows the GPU that the browser uses.
Browser that an organization controls Chrome · Edge
An organization can set graphics acceleration to off with a policy. Then the browser menu shows Managed by your organization. To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
- Open the address
chrome://policy(in Edge,edge://policy). - Find the policy
HardwareAccelerationModeEnabled. - If its value is
false, tell your administrator that the app must have graphics acceleration.
Firefox Firefox
- Windows: install Firefox, version 141 or newer.
- Mac with Apple silicon: install Firefox, version 147 or newer.
- Mac with an Intel processor, Linux, and Android: Firefox has no WebGPU at this time. Use Chrome or Edge.
Safari Safari
Safari has WebGPU from version 26, on macOS 26, iOS 26, and iPadOS 26.
- Update the operating system to version 26 or newer.
- If you cannot update it, use Chrome or Edge on a computer.
The page is not a secure page
WebGPU operates only on a secure page. The address of a secure page starts with https://.
Open the secure address
- Open the address
https://app.tracinginsights.com/chat. - Make sure that the address bar shows
https://.
On your computer, an address that starts with http://localhost is also a secure page.
The browser gives no GPU to WebGPU
The browser has WebGPU, but it gave no GPU to this page. The usual causes are these: graphics acceleration is off, the browser does not accept the GPU driver, or Vulkan is off (Linux).
Linux: set Vulkan to on in Chrome Linux · Chrome
On Linux, Chrome uses Vulkan for WebGPU. On many Linux computers, Chrome has Vulkan set to off. Then WebGPU finds no GPU. On the test laptop of the developers (Chrome 157, Wayland, Intel and NVIDIA GPUs), only this flag was necessary. To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
CAUTION: Change only the flag that this procedure gives. Chrome shows the flags as Experiments, and an incorrect flag can stop the browser.
Open the addresschrome://flags/#enable-vulkan.- Set Vulkan to Enabled.
- Click Relaunch.
- Open the address
chrome://gpu. - Make sure that the line Vulkan shows Enabled.
- Open this app again.
Linux: set the WebGPU flag in Chrome Linux · Chrome
Do this procedure only if the message stays after the Vulkan procedure.
CAUTION: Change only the flag that this procedure gives. Chrome shows the flags as Experiments, and an incorrect flag can stop the browser.
Open the addresschrome://flags/#enable-unsafe-webgpu.- Set Unsafe WebGPU Support to Enabled.
- Click Relaunch.
- Open this app again.
Linux: install the NVIDIA driver Linux
For an NVIDIA GPU, use the NVIDIA proprietary driver, version 535.183 or newer. For versions before 535.183, Chrome has WebGPU set to off by default.
- Install the NVIDIA driver package of your Linux distribution.
- Start the computer again.
- Type this command in a terminal and press Enter:
nvidia-smi - Make sure that the result shows your GPU and the driver version.
Set graphics acceleration to on in Chrome Chrome
To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
- Open the address
chrome://settings/system. - Set Use graphics acceleration when available to on.
- Click Relaunch.
- Open this app again.
Set graphics acceleration to on in Edge Edge
To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
- Open the address
edge://settings/system. - Set Use graphics acceleration when available to on.
- Close all Edge windows, then start Edge again.
- Open this app again.
Read the GPU report of the browser Chrome · Edge
To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
- Open the address
chrome://gpu(in Edge,edge://gpu). - Find the line WebGPU in the section Graphics Feature Status.
- Compare the text of the line with this list.
- Hardware accelerated: WebGPU can use the GPU.
- Software only, hardware acceleration unavailable: graphics acceleration is off, or the browser does not accept the GPU driver.
- Disabled or Unavailable: the browser set WebGPU to off. The section Problems Detected gives the cause.
The lines GPU0 and GPU1 show the GPUs of the computer. The text *ACTIVE* shows the GPU that the browser uses.
Windows: install the newest GPU driver Windows
The browser has a list of GPU drivers that have known problems. It does not use the GPU with such a driver.
- In Windows 11, open Settings > Windows Update > Advanced options > Optional updates.
- Select the driver updates.
- Click Download and install.
- Start the computer again.
- Open this app again.
In Windows 10, the path is Settings > Update & Security > Windows Update > View optional updates. The GPU manufacturer also has drivers: nvidia.com/drivers, amd.com/support, or the Intel driver page.
Linux: update the GPU driver Linux
- Update the system with the package manager of your distribution.
- Start the computer again.
- Open this app again.
Do a test in the browser console Chrome · Edge
- Open this app in a new tab.
- Press F12, then select Console.
- Type this line and press Enter:
(await navigator.gpu.requestAdapter({ powerPreference: 'high-performance' }))?.info - Compare the result with this list.
nullorundefined: the browser gives no GPU to the page.- An object with
vendorandarchitecture: the browser gives that GPU to the page.
Remote desktop and virtual machines
In a remote desktop session or a virtual machine, the browser frequently has no GPU.
- Use the app in a browser on the computer that has the GPU.
WebGPU uses the processor, not the GPU
The browser gave a software renderer to this page (SwiftShader, llvmpipe, or WARP). A software renderer does the GPU work on the processor. The model is too slow on a software renderer, and the app does not load it.
Set graphics acceleration to on in Chrome Chrome
To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
- Open the address
chrome://settings/system. - Set Use graphics acceleration when available to on.
- Click Relaunch.
- Open this app again.
Set graphics acceleration to on in Edge Edge
To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
- Open the address
edge://settings/system. - Set Use graphics acceleration when available to on.
- Close all Edge windows, then start Edge again.
- Open this app again.
Linux: set Vulkan to on in Chrome Linux · Chrome
On Linux, Chrome uses Vulkan for WebGPU. On many Linux computers, Chrome has Vulkan set to off. Then WebGPU finds no GPU. On the test laptop of the developers (Chrome 157, Wayland, Intel and NVIDIA GPUs), only this flag was necessary. To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
CAUTION: Change only the flag that this procedure gives. Chrome shows the flags as Experiments, and an incorrect flag can stop the browser.
Open the addresschrome://flags/#enable-vulkan.- Set Vulkan to Enabled.
- Click Relaunch.
- Open the address
chrome://gpu. - Make sure that the line Vulkan shows Enabled.
- Open this app again.
Read the GPU report of the browser Chrome · Edge
To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
- Open the address
chrome://gpu(in Edge,edge://gpu). - Find the line WebGPU in the section Graphics Feature Status.
- Compare the text of the line with this list.
- Hardware accelerated: WebGPU can use the GPU.
- Software only, hardware acceleration unavailable: graphics acceleration is off, or the browser does not accept the GPU driver.
- Disabled or Unavailable: the browser set WebGPU to off. The section Problems Detected gives the cause.
The lines GPU0 and GPU1 show the GPUs of the computer. The text *ACTIVE* shows the GPU that the browser uses.
Windows: install the newest GPU driver Windows
The browser has a list of GPU drivers that have known problems. It does not use the GPU with such a driver.
- In Windows 11, open Settings > Windows Update > Advanced options > Optional updates.
- Select the driver updates.
- Click Download and install.
- Start the computer again.
- Open this app again.
In Windows 10, the path is Settings > Update & Security > Windows Update > View optional updates. The GPU manufacturer also has drivers: nvidia.com/drivers, amd.com/support, or the Intel driver page.
Linux: update the GPU driver Linux
- Update the system with the package manager of your distribution.
- Start the computer again.
- Open this app again.
Remote desktop and virtual machines
In a remote desktop session or a virtual machine, the browser frequently has no GPU.
- Use the app in a browser on the computer that has the GPU.
The GPU cannot operate the model
The GPU or its driver does not have a function that the model must have. The text in parentheses in the message gives the cause.
Computer with two GPUs
A laptop frequently has an integrated GPU and a dedicated GPU. The browser can give the integrated GPU to the page.
- Do the procedures in Use the dedicated GPU.
Windows: install the newest GPU driver Windows
The browser has a list of GPU drivers that have known problems. It does not use the GPU with such a driver.
- In Windows 11, open Settings > Windows Update > Advanced options > Optional updates.
- Select the driver updates.
- Click Download and install.
- Start the computer again.
- Open this app again.
In Windows 10, the path is Settings > Update & Security > Windows Update > View optional updates. The GPU manufacturer also has drivers: nvidia.com/drivers, amd.com/support, or the Intel driver page.
Linux: update the GPU driver Linux
- Update the system with the package manager of your distribution.
- Start the computer again.
- Open this app again.
Update Chrome Chrome
- In the Chrome menu, select Help > About Google Chrome.
- Wait until Chrome completes the update.
- If Relaunch shows, click it.
On Linux, update Chrome with the package manager of your distribution.
Update Edge Edge
- In the Edge menu (Settings and more), select Help and feedback > About Microsoft Edge.
- Wait until Edge completes the update.
- If Restart shows, click it.
If the message stays
Then this browser or this GPU cannot operate the model. The chat continues to answer with your other provider.
- Select a different provider in the provider list of the chat.
Use the dedicated GPU, not the integrated GPU
A laptop frequently has two GPUs: an integrated GPU (low power, slow) and a dedicated GPU (fast). The app always tells the browser to use the high-performance GPU. But the browser and the operating system make the decision. The model is much faster on the dedicated GPU.
Find the GPU that the page uses
- In this app, open Settings > AI Provider > On-device model.
- If Check this browser shows, click it.
- Read the text after GPU: in the status box.
- NVIDIA: the page uses the NVIDIA GPU. In a laptop, that is the dedicated GPU.
- Intel: in a laptop with two GPUs, that is usually the integrated GPU.
- AMD: AMD makes integrated GPUs and dedicated GPUs. Compare the name with the GPU report of the browser.
After a load, Last load in the same panel also shows the GPU.
Read the GPU report of the browser Chrome · Edge
To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
- Open the address
chrome://gpu(in Edge,edge://gpu). - Find the line WebGPU in the section Graphics Feature Status.
- Compare the text of the line with this list.
- Hardware accelerated: WebGPU can use the GPU.
- Software only, hardware acceleration unavailable: graphics acceleration is off, or the browser does not accept the GPU driver.
- Disabled or Unavailable: the browser set WebGPU to off. The section Problems Detected gives the cause.
The lines GPU0 and GPU1 show the GPUs of the computer. The text *ACTIVE* shows the GPU that the browser uses.
Windows: set the browser to High performance Windows
On Windows, Chrome and Edge use the GPU that Windows gives to the browser. There, the GPU preference of the app has no effect. When this Windows setting is set, the NVIDIA and AMD control panels have no effect.
- Open Settings > System > Display > Graphics.
- Find the browser in the list of apps.
- If the browser is not in the list, add its program file (
chrome.exeormsedge.exe). - Select the browser. In some Windows versions, click Options.
- Select High performance.
- If Save shows, click it.
- Close all browser windows, then start the browser again.
The names on this page are different between Windows versions. In Windows 10, the path is Settings > System > Display > Graphics settings. To find the program file, open the address chrome://version and read Executable Path.
Windows: set the high-performance GPU flag in Chrome Windows · Chrome
Do this procedure if the page uses the integrated GPU after the Windows procedure. To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
CAUTION: Change only the flag that this procedure gives. Chrome shows the flags as Experiments, and an incorrect flag can stop the browser.
Open the addresschrome://flags/#force-high-performance-gpu.- Set Force High Performance GPU to Enabled.
- Click Relaunch.
Windows: NVIDIA Control Panel Windows
From Windows 10 version 2004, this panel has no effect when Windows has a GPU setting for the browser. Do this procedure only if Windows has no GPU setting for the browser.
- Open NVIDIA Control Panel.
- Select Manage 3D Settings, then the tab Program Settings.
- In Select a program to customize, select the browser. If it is not in the list, click Add.
- In Preferred graphics processor, select the high-performance NVIDIA processor.
- Apply the change.
- Close all browser windows, then start the browser again.
Windows: AMD GPU Windows
AMD Software has no setting for this from version 19.12.1. AMD tells you to use the Windows setting.
- Do the procedure "Windows: set the browser to High performance".
Laptop: connect the power supply
On battery power, some laptops give the integrated GPU to the browser.
- Connect the laptop to its power supply.
- Close all browser windows, then start the browser again.
Mac with two GPUs macOS
A Mac with Apple silicon has one GPU. No setting is necessary there. Some MacBook Pro models with an Intel processor have two GPUs.
- Select the Apple menu > System Settings > Battery.
- Click Options.
- Set Automatic graphics switching to off.
- Close the browser, then start it again.
Then the Mac always uses the dedicated GPU, and the battery becomes empty more quickly. Before macOS 13, the setting is in System Preferences > Battery. If the setting does not show, the Mac has only one GPU.
Linux: laptop with two GPUs Linux
With Vulkan on, Chrome gave the NVIDIA GPU to the app on the test laptop of the developers. If the page uses the integrated GPU, start Chrome with the NVIDIA GPU first in the Vulkan list.
- Close all Chrome windows.
- Type this command in a terminal and press Enter:
__NV_PRIME_RENDER_OFFLOAD=1 __VK_LAYER_NV_optimus=NVIDIA_only google-chrome-stable
- Arch Linux: the package
nvidia-primehas the commandprime-run. Useprime-run google-chrome-stable. - Ubuntu:
sudo prime-select nvidiasets the NVIDIA GPU for all programs. Start the computer again after the command. - Mesa driver (AMD or Intel dedicated GPU): use
DRI_PRIME=1 google-chrome-stable. - GNOME: right-click the Chrome icon and select Launch using Discrete Graphics Card.
These environment variables come from the NVIDIA and Mesa driver documentation. We did no test of them with Chrome.
The GPU memory is not sufficient
The model uses about 1.4 GB of GPU memory. The load stops when the GPU does not have that quantity of available memory. Other tabs and other programs use the same GPU memory.
Make GPU memory available
- Close other tabs that use the GPU (games, video, 3D maps, other AI models).
- Close other programs that use the GPU.
- Click Retry.
Windows: find the programs that use the GPU Windows
- Press Ctrl+Shift+Esc to open Task Manager.
- Select Performance, then select the GPU.
- Read the memory values of the GPU.
- Select Processes to find the programs that use the GPU.
Start the browser again
A browser that operated for a long time can keep GPU memory that it does not use.
- Close all browser windows.
- Start the browser again.
- Open the app and click Load.
Computer with two GPUs
An integrated GPU frequently has less available memory than a dedicated GPU.
- Do the procedures in Use the dedicated GPU.
The model is slow
The model is as fast as the GPU that it uses. A dedicated GPU is much faster than an integrated GPU. Other GPU work on the computer decreases the speed.
Make sure that the page uses the dedicated GPU
- Do the procedures in Use the dedicated GPU.
Laptop: connect the power supply
On battery power, some laptops give the integrated GPU to the browser.
- Connect the laptop to its power supply.
- Close all browser windows, then start the browser again.
Stop other GPU work
- Close games, video, and other AI models.
- Send the question again.
Keep the GPU active while you type
A desktop GPU decreases its speed some seconds after its last work. Then the first answer after some time without questions starts slowly.
- In this app, open Settings > AI Provider > On-device model.
- Set Prime the GPU while you type to On.
The setting Auto has no effect when the device uses battery power.
Browser without Web Workers
Without Web Workers, the model and the page use the same thread. Then the page can stop for short times while the model answers.
- Install the newest version of Chrome or Edge.
The storage space is not sufficient
The model files use about 1.15 GB of the storage that the browser gives to this site. The browser gives each site a part of the disk space. The message shows the space that is necessary and the space that the browser gives.
Make disk space available
- Remove files that you do not use from the disk.
- Make sure that the disk has more than 2 GB of available space.
- In this app, click Check again in Settings > AI Provider > On-device model.
Do not use a private window
In an Incognito or InPrivate window, the storage limit is much smaller.
- Open the app in a window that is not a private window.
Chrome: remove the data of other sites Chrome
- Open Settings > Privacy and security > Third-party cookies > See all site data and permissions.
- Find a site that keeps much data and that you do not use.
WARNING: Make sure that you can sign in to that site again. This step removes its sign-in and all its data.
Click the delete icon of the site, then click Delete.- In this app, click Check again in Settings > AI Provider > On-device model.
Edge: remove the data of other sites Edge
- Open Settings and more > Settings > Privacy, search, and services > Cookies > See all cookies and site data.
- Find a site that keeps much data and that you do not use.
WARNING: Make sure that you can sign in to that site again. This step removes its sign-in and all its data.
Click the arrow of the site, then click Delete.- In this app, click Check again in Settings > AI Provider > On-device model.
Chrome: setting that deletes site data Chrome
Chrome can delete the data of all sites when you close all windows. With that setting, the storage limit is about 300 MB. This is too small for the model. To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
- Open the address
chrome://settings/content/siteData. - Select the setting that lets sites keep data on your device.
- In this app, click Check again in Settings > AI Provider > On-device model.
The browser gives no storage to this site
The model files stay in the storage that the browser gives to this site. This message shows when the browser gives no storage. The usual cause is a private window.
Do not use a private window
- Open the app in a window that is not a private window.
Firefox gives no file storage to sites in Private Browsing. Safari can do the same in Private Browsing.
Chrome: let sites keep data Chrome
To open an address that starts with chrome:// or edge://, type it in the address bar and press Enter.
- Open the address
chrome://settings/content/siteData. - Select the setting that lets sites keep data on your device.
- In this app, click Check again in Settings > AI Provider > On-device model.
Update Chrome Chrome
- In the Chrome menu, select Help > About Google Chrome.
- Wait until Chrome completes the update.
- If Relaunch shows, click it.
On Linux, update Chrome with the package manager of your distribution.
Update Edge Edge
- In the Edge menu (Settings and more), select Help and feedback > About Microsoft Edge.
- Wait until Edge completes the update.
- If Restart shows, click it.
The browser can delete the download
When the disk is almost full, the browser can delete the files of a site. Then the model downloads again (1.14 GB). With persistent storage, the browser keeps the files. The app tells the browser to give persistent storage when you start the download or the load.
Chrome and Edge Chrome · Edge
Chrome and Edge make the decision. They show no question. They give persistent storage to a site in one of these conditions:
- You installed the site as an app.
- The site has a bookmark.
- The site has permission to show notifications.
- You use the site frequently.
- Click Install app in the status of the model.
- If Install app does not show, add a bookmark for this page.
- Start the download or the load again.
Then the app tells the browser again to give persistent storage.
Firefox Firefox
Firefox shows a question when you click Download or Load. The question starts with Allow and ends with to store data in persistent storage?
- Click Allow.
If you click Block, Firefox does not show the question again for some time. In Private Browsing, Firefox gives no persistent storage.
Safari Safari
Safari makes the decision. It shows no question. A site that you open from the Home Screen or the Dock can get persistent storage.
If the browser deleted the download
- Make more disk space available.
- Click Download.
- Do the procedure for your browser in this section.
The download stopped
The model files come from Hugging Face (huggingface.co). The download has parts of 32 MiB, and the app examines each part. After a stop, the download continues from the last complete part.
Continue the download
- Make sure that the computer has an internet connection.
- Click Retry or Resume download.
The network does not let the download through
A firewall, a content filter, or a VPN can stop the connection to Hugging Face.
- Open the address
https://huggingface.coin a new tab. - If the page does not open, use a different network, or set the VPN or the filter to off.
- Click Retry.
A part of the download has damage
- Click Retry.
The app downloads the damaged part again. If the message shows again, remove the download and start it again:
- In this app, open Settings > AI Provider > On-device model.
- Click Remove downloaded model.
- Click Download.
The app cannot read the file list
The file list tells the app which model files to download. Only the newest version of the app can read a new file list.
- Load the page again to get the newest version of the app.
- Click Retry.
- If the message stays, do the procedure again after some hours.
A different tab has the model
Only one tab can use the model at a time. The model uses much GPU memory, and one more copy does not help.
Use the model in this tab
- Click Use it here.
The model stops in the other tab and loads in this tab. Questions in the other tab then use your other provider.
Use the tab that has the model
- Go to the other tab and send your question there.
The model stopped
The browser stopped the GPU work of this page, or the load did not complete. This can occur after sleep mode, after a driver update, or when the GPU has too much work. The text in parentheses in the message is the error text from the browser.
Load the model again
- Click Load or Retry.
The model loads from the storage of the browser in some seconds. It does not download again.
The model stopped two times in 10 minutes
After two stops in 10 minutes, the app sets the model to off for this page.
- Close other tabs and programs that use the GPU.
- Load the page again.
- Click Load.
Start the browser again
- Close all browser windows.
- Start the browser again.
- Open the app and click Load.
Windows: install the newest GPU driver Windows
The browser has a list of GPU drivers that have known problems. It does not use the GPU with such a driver.
- In Windows 11, open Settings > Windows Update > Advanced options > Optional updates.
- Select the driver updates.
- Click Download and install.
- Start the computer again.
- Open this app again.
In Windows 10, the path is Settings > Update & Security > Windows Update > View optional updates. The GPU manufacturer also has drivers: nvidia.com/drivers, amd.com/support, or the Intel driver page.
Linux: update the GPU driver Linux
- Update the system with the package manager of your distribution.
- Start the computer again.
- Open this app again.
If the message stays
- Copy the full text of the message.
- Send it to the developers on the page Feedback.
No model for this version of the app
We make each on-device model for one version of the instructions that the app sends to it. After an app update, the model for the new instructions is not always available immediately. No setting on your computer can change this.
Use a different provider until the model is available
- Select a different provider in the provider list of the chat.
- Do the check again after some days in Settings > AI Provider > On-device model.
The conversation is too long for the model
The on-device model reads a maximum of 20,480 tokens for each question. These tokens contain the instructions of the app, the conversation, and your question.
Make the conversation shorter
- Start a new conversation.
- If the message shows again, write a shorter question.
The local model with llama-server
You can also operate the same model with llama-server from llama.cpp. Then the model does not use the browser. You add the server to the app as a custom provider. These procedures are for persons who know the command line.
Get llama.cpp
Use a llama.cpp release from 2026-10-27 or newer. Releases before that date cannot read this model (LFM3).
- Open the address
https://github.com/ggml-org/llama.cpp/releases. - Download the file for your computer from the newest release.
- Copy the contents of the downloaded file into a new folder.
- Windows with an NVIDIA GPU: the two files with
win-cudain their names. - Windows with a different GPU: the file with
win-vulkanin its name. - Linux: the file with
ubuntu-vulkanin its name, orubuntu-cudafor an NVIDIA GPU. - Mac: the file with
macosin its name. On a Mac, llama.cpp uses the GPU through Metal.
Start the server
- Open a terminal in the folder that contains
llama-server. - Type this command on one line and press Enter:
llama-server -hf tracinginsights/tif1-LFM3-1.7B-GGUF --hf-file tif1-LFM3-1.7B-v1.1-Q4_0.gguf --alias tif1-lfm3-q4 --jinja -np 4 -c 81920 --ctx-checkpoints 16 -fa on --slot-save-path ~/.cache/tif1/slots --host 127.0.0.1 --port 8080 - Wait until the terminal shows a line that contains
listening on.
The first start downloads the model file (1.05 GB) from Hugging Face. If you have the file, use -m and the path of the file as an alternative to -hf and --hf-file.
--alias tif1-lfm3-q4: the app identifies the model by this name. The name must start withtif1-lfm3.-np 4: 4 slots, one for each type of question that the app sends.-c 81920: 20,480 tokens for each slot. The server divides the context between the slots.--jinjaand--ctx-checkpoints 16: the server keeps the long instructions of the app in memory between questions.-fa on: Flash Attention. It makes the server faster on a GPU.--slot-save-path ~/.cache/tif1/slots: the folder where the server can keep the slots.--host 127.0.0.1: only programs on your computer can connect to the server.--port 8080: keep this option. New llama.cpp releases use a different port by default.
Add the server to the app
- In this app, open Settings > AI Provider > Custom providers.
- Click Add.
- Type a name in Label.
- Select OpenAI Chat Completions in Protocol.
- Type
http://127.0.0.1:8080/v1in Base URL. - Click Auto-discover. The model
tif1-lfm3-q4shows in Models. - Click Save.
- In On-device model, set Where the tif1 local model runs to llama-server (a custom provider).
- Select the new provider in the provider list of the chat.
If the browser shows a question about a connection to devices on your local network, click Allow.
Make llama.cpp use the dedicated GPU
By default, llama.cpp puts all layers of the model on the GPU. The Vulkan release does not use the integrated GPU when the computer has a dedicated GPU.
- To see the GPUs that llama.cpp finds, type this command and press Enter:
llama-server --list-devices - Find the name of the dedicated GPU at the start of its line.
- Add
--deviceand that name to the start command. - Add
-ngl 99to the start command to put all layers on the GPU. - Start the server again.
- Find the line that contains
layers to GPUin the terminal. - Make sure that the two numbers in the line are equal.
- CUDA release: the variable
CUDA_VISIBLE_DEVICESselects the NVIDIA GPU by its number, for exampleCUDA_VISIBLE_DEVICES=0. - Vulkan release: the variable
GGML_VK_VISIBLE_DEVICESselects the GPU by its number, for exampleGGML_VK_VISIBLE_DEVICES=0. - Mac: Metal uses the default GPU of the Mac.
--device nonesets the GPU to off. - The text
no usable GPU foundshows that this llama.cpp file has no GPU function. Download the CUDA or Vulkan file.
If the server is slow
- Make sure that all layers are on the dedicated GPU (the procedure before this one).
- Make sure that the start command contains
-fa on. - Close other programs that use the GPU.
- Connect the laptop to its power supply.
The first question after a server start is slow. The server reads the long instructions of the app one time for each slot.
If the server does not start
Find one of these texts in the terminal:
couldn't bind HTTP server socket: a different program uses the port. Stop that program, or change--portin the command and in Base URL.unknown model architecture: this llama.cpp release cannot read the model. Download the newest release.failed to load model: the model file is not complete, or its path is incorrect. Download the file again.cudaMalloc failed: the GPU memory is not sufficient. Close other programs that use the GPU.
- To do a test, open the address
http://127.0.0.1:8080/healthin the browser. - Make sure that the page shows the status
ok.
The text Loading model shows that the model load is not complete. Wait, then load the page again.
If the app shows a context message
The model must have 20,480 tokens for each slot. The options -c 20480 -np 4 give only 5,120 tokens to each slot. That is too small.
- Stop the server.
- Start it again with
-c 81920 -np 4in the command.