Take the slides with you

Blablador

/ˈblæblæˌdɔɹ/
Bla-bla-bla 🗣️ + Labrador 🐕‍🦺
A stage for deploying and testing large language models
Models change constantly (constantly improving rank, some good, some awful)
Usually a small/fast model and fone of the top of the HF’s Open LLM Leaderboard
It is a web server and an api server, and training code.

“I think the complexity of Python package management holds down AI application development more than is widely appreciated. AI faces multiple bottlenecks — we need more GPUs, better algorithms, cleaner data in large quantities. But when I look at the day-to-day work of application builders, there’s one additional bottleneck that I think is underappreciated: The time spent wrestling with version management is an inefficiency I hope we can reduce.”

Andrew Ng, 28.02.2024

“Building on top of open source can mean hours wrestling with package dependencies, or sometimes even juggling multiple virtual environments or using multiple versions of Python in one application. This is annoying but manageable for experienced developers, but creates a lot of friction for new AI developers entering our field without a background in computer science or software engineering.”

Andrew Ng, 28.02.2024

Why?

AI is becoming basic infrastructure
Which historically is Open Source
We train a lot, deploy little: Here is your code/weights, tschüssi!
Little experience with dealing with LLMs
From the tools point of view, this is a FAST moving target 🎯💨
Acquire local experience in issues like
- data loading,
- quantization,
- distribution,
- fine-tune LLMs for specific tasks,
- inference speed,
- deployment
Projects like OpenGPT-X, TrustLLM and Laion need a place to run
The usual: we want to be ready when the time comes
TL;DR: BECAUSE WE CAN! 🤘

Some facts

We have no models of our own (yet) deployed
Can host one model with 8 gpus, and multiple with smaller size (Haicluster is 16 gpus)
Mixtral-8x7b: takes 7 gpus
No data collection at all. I don’t keep ANY data whatsoever
- You can use it AND keep your data private
- No records? Privacy (and GDPR is happy)

Deployment as a service

Scientists from (currently just FZJ) can deploy their models on their hardware and point to blablador
This solves a bunch of headaches for researchers:
- Authentication
- Web server
- Firewall
- Availability
- Etc
If you have a model and want to deploy it, contact me!

OpenAI-compatible API

Uses openai-python from OpenAI itself
All services which can use OpenAI’s API can use Blablador’s API (VSCode’s Continue.dev, etc)
The API is not yet rate-limited, logged, monitored, documented or well-tested.

Website

https://helmholtz-blablador.fz-juelich.de

Play around! 🐶

API Access

Create a token in the helmholtz codebase settings - (https://codebase.helmholtz.cloud) with the api scope
Copy the token
Go to https://helmholtz-blablador.fz-juelich.de:8000/docs
Click on Authorize, and paste the token

Demo: API

Go to /v1/models
Click on Try it out
Click on Execute

Demo: API

Demo: cURL

curl --header "Authorization: Bearer MY_TOKEN_GOES_HERE"   https://helmholtz-blablador.fz-juelich.de:8000/v1/models

Demo: VScode + Continue.dev

Yes. It DOES run with Emacs too. Ask your favorite Emacs expert.
Add continue.dev extension to VSCode
On Continue, choose to add model, choose Other OpenAI-compatible API
Click in Open Config.json at the end

Demo: VScode + Continue.dev

Inside config.json, add at the "models" section:

    {
      "title": "Mistral helmholtz",
      "provider": "openai",
      "contextLength": 16384,
      "model": "Mistral-7B-Instruct-v0.2",
      "apiKey": "YOUR_TOKEN_GOES_HERE",
      "apiBase": "https://helmholtz-blablador.fz-juelich.de:8000"
    },