No Comments

Running the Dutch LLM fietje-2-chat with reasonable speed on your Android Phone

{,}
October 10th, 2024

To run Fietje-2-Chat on your Android Phone locally on a reasonable speed, you’ll need to create a special quantized version of fietje-2b-chat.

One of the best apps to run a LLM’s on your Android Phone is ChatterUI (https://github.com/Vali-98/ChatterUI).

You can download the APK from Github and transfer it to your phone and install it. It’s not yet available in F-Droid.

As most Android Phones have a ARM CPU, use special `quants` that run faster on ARM, because the use NEON extensions, int8mm and SVE instructions.

Note that these optimized kernels require the model to be quantized into one of the formats: Q4_0_4_4 (Arm Neon), Q4_0_4_8 (int8mm) or Q4_0_8_8 (SVE). The SVE mulmat kernel specifically requires a vector width of 256 bits. When running on devices with a different vector width, it is recommended to use the Q4_0_4_8 (int8mm) or Q4_0_4_4 (Arm Neon) formats for better performance.

https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md#arm-cpu-optimized-mulmat-kernels

How to create a special ARM optimized version of Fietje-2-Chat

Download the f16 guff version of Fietje-2-Chat:

wget https://huggingface.co/BramVanroy/fietje-2-chat-gguf/resolve/main/fietje-2b-chat-f16.gguf?download=true

Install a Docker version of LLama to do the conversion

mkdir p ~/llama/models
sudo docker run -v /home/user/llama/models:/models ghcr.io/ggerganov/llama.cpp:full --all-in-one "/models/" 7B

To convert the f32 or f16 gguf to another format Q4_0_4_4 (Arm Neon):

docker run --rm -v /home/user/llama/models:/models ghcr.io/ggerganov/llama.cpp:full --quantize "/models/fietje-2b-chat-f16.gguf" "/models/fietje-2b-chat-Q4_0_4_4.gguf" "Q4_0_4_4"

Transfer the fietje-2b-chat-Q4_0_4_4.gguf to your Android Device.

Open ChatterUI

Import Model:

Go to menu -> API -> Model -> import -> fietje-2b-chat-Q4_0_4_4.gguf

Load Model:

menu ->API -> Model -> select fietje-2b-chat-Q4_0_4_4.gguf -> Load

Then leave the settings and start typing in the prompt. The first cold run will be a little slow, but once it’s running you’ll get about 10 tokens/s on a Snapdragon 865 phone.

That’s not bad.

If you’re interested in a LLM that can generate much better Dutch than LLama3.2 or Phi3 on your phone, give Fietje a try.

No Comments

Creating graphics with LLM’s: generate SVG!

{,}
September 25th, 2024

LLM’s can generate text, and Scalable Vector Graphics (SVG) is an XML-based vector image format for defining two-dimensional graphic. XML is human readable, so it can be generated by LLM’s.

How good do they perform?

https://duckduckgo.com/?q=DuckDuckGo+AI+Chat&ia=chat&duckai=1

Generate a dog in svg

Looks like a cute mix between a rubber duck and a dog to me.

Generate a chair in svg

More a stool then chair.

"A mix between a rubber duck and a dog."

LLM’s have humor in a way.

No Comments

Open source coding with Ollama as Copilot

{,,}
September 24th, 2024

Among the many tools available to developers today, support from AI as a coding assistant is a nice must-have.

Zed is a Rust-written modern open source code editor designed for collaborative editing and multiplayer teamwork. It works fine as a stand-alone editor with git support.

Ollama offers easy and privacy-friendly local driven LLM support. Get up and running with large language models on your own machine.

Zed does offer AI integration with ChatGPT or Claude, but it can also connect to your local Ollama install.

To try this out, just add this to the settings-file in Zed CTRL + ,:

 

"assistant": {
"version": "1",
"provider": {
"default_model": {
"name": "qwen2.5-coder",
"display_name": "codeqwen",
"max_tokens": 2048,
"keep_alive": -1
},
"name": "ollama",
// Recommended setting to allow for model startup
"low_speed_timeout_in_seconds": 30
}
}

Open a new assistant tab and you can change context of the assistant to your tab content, and let the AI assistant annotate your script:

/tab
Annotate the JS file

And can LLM’s create graphics? Of course they can.

How to create images with a Large Language Model (LLM)

SVG!

SVG is an xml based format that can be understood by humans and machines, so when a coding assistant can write code, it can also write SVG. 😉

Does it create sensible graphics? Not really.

I asked qwen2.5-coder  in Zed:

Create a SVG file that shows a chair.

Does this look like a chair?

 

Another attempt:

Does it really look like a chair? What does it look like. Let me know in the comments!

No Comments

Configuring Tmux for better handling

August 6th, 2024

Tmux is a terminal multiplexer that enables the concurrent execution of multiple terminal programs within a single terminal session.

This feature significantly enhances productivity by allowing users to monitor various logs, debug programs, settings, and network traffic simultaneously.

For further information on Tmux’s capabilities and benefits, please refer to the documentation.

While Tmux offers a range of customizable settings, some default configurations may require adjustment.

Specifically, the default keyboard shortcut for the Command Prefix, CTRL+B, may be inconvenient for users who prefer to use one hand.

To address this, I recommend modifying the shortcut to CTRL+Q.

Additionally, Tmux’s scrolling functionality can be improved. I already mapped the CapsLk key to a CTRL key in Ubuntu/Gnome.

Scrolling is also not that trivial. So we gonna add mouse/trackpad scrolling. That’s a nice alternative to the keyboard way: Command Prefix + } followed by one of the cursor keys.

Edit .tmux.conf and put the following in:

unbind C-b
set-option -g prefix C-q
set-option -g mouse That's all for now.
No Comments

How to fix the no header and footer showing in Woocommerce and WordPress 2024 child themes

August 3rd, 2024

Not really a surprise, we saw it coming for several years, and it’s not necessarily a bad thing lowering technology barriers for WordPress users, but in practice what is the gain of multiple inline CSS rules, instead of one properly made external CSS sheet.

Similarly, fixing the wrongly collapsing navigation menu on mobile is just a matter of adjusting a number in a media query, but there is no fucking way to do that in the UI.

What’s the gain of writing myriads of JavaScript rules to be able to change a CSS variable, which you could do by simply changing a variable in the CSS file. It’s a loss for the developers and a gain for the noobs, the marketeers and the trackers. It’s like taking your car the drive to your postbox to empty it.

But it’s an efficiency loss for the web in general. Pushing and forcing JavaScript for these things is simply wrong.

Anyhow when you develop a new site with the new 2024 theme, you can get far by clicking and dragging your mouse, but I can assure you, when you do that a couple of times and tried out several themes, headers, layout and patterns, there comes a moment headers and footers are lost or different on the WooCommerce pages.

Why and how you should fixed this with the UI/ Visual Editor, is quite obscure, but using the code editor, it’s a matter of fixing the first and last row in a WooCommerce page:

Do a site edit and not a page edit on a Woocommerce page and enable the code editor in the right top settings and change this

<!-- wp:template-part {"slug":"header","theme":"twentytwentyfour"} /--> to

<!-- wp:template-part {"slug":"header","theme":"<your_theme_name>"} /-->

Vice versa for the footer.

I don’t know why the checkout page in WooCommerce has a different header the the other shop pages, (and no footer by default) but simply change this

<!-- wp:template-part {"slug":"checkout-header","theme":"woocommerce/woocommerce","tagName":"header"} /-->

to

<!-- wp:template-part {"slug":"header","theme":"<your_theme_name>"} /-->

No Comments

Fixing No Audio on Kodi 21 Flatpak on Ubuntu 22.04 LTS

{}
April 17th, 2024

Somehow my new Flatpak Kodi 21 install stopped outputting any sound after an update.

As it seems the Flatpak update is now by default outputting only PipeWire sound, (or with higher priority than before) and that doesn’t work on Ubuntu 22.04 LTS

To fix it, simply run Kodi with --audio-backend=pulseaudio:

 flatpak run tv.kodi.Kodi --audio-backend=pulseaudio

That’s all. Sound is working again on Ubuntu 22.04.

I guess this will be fixed on another Kodi Flatpak update.

See the issue here:

https://github.com/xbmc/xbmc/issues/25006