Website

Blablador

/ˈblæblæˌdɔɹ/
Bla-bla-bla 🗣️ + Labrador 🐕‍🦺
A stage for deploying and testing large language models
Models change constantly (constantly improving rank, some good, some awful)
Usually a small/fast model and fone of the top of the HF’s Open LLM Leaderboard
It is a web server and an api server, and training code.

“I think the complexity of Python package management holds down AI application development more than is widely appreciated. AI faces multiple bottlenecks — we need more GPUs, better algorithms, cleaner data in large quantities. But when I look at the day-to-day work of application builders, there’s one additional bottleneck that I think is underappreciated: The time spent wrestling with version management is an inefficiency I hope we can reduce.”

Andrew Ng, 28.02.2024

“Building on top of open source can mean hours wrestling with package dependencies, or sometimes even juggling multiple virtual environments or using multiple versions of Python in one application. This is annoying but manageable for experienced developers, but creates a lot of friction for new AI developers entering our field without a background in computer science or software engineering.”

Andrew Ng, 28.02.2024

Why?

AI is becoming basic infrastructure
Which historically is Open Source
We train a lot, deploy little: Here is your code/weights, tschüssi!
Little experience with dealing with LLMs
From the tools point of view, this is a FAST moving target 🎯💨
Acquire local experience in issues like
- data loading,
- quantization,
- distribution,
- fine-tune LLMs for specific tasks,
- inference speed,
- deployment
Projects like OpenGPT-X, TrustLLM and Laion need a place to run
The usual: we want to be ready when the time comes
TL;DR: BECAUSE WE CAN! 🤘

Some facts

We have no models of our own (yet) deployed
Models based on Llama2-70 🦙 take 7 gpus (or 8 with vLLM)
VLLM: PagedAttention, batching etc. Speeds up inference at cost of gpus (no quantization)
SGLang: RadixAttention - Even faster inference. Single-GPU for now
Mixtral-8x7b: takes 7 gpus, but is faster and better than Llama2-70
No data collection at all. I don’t keep ANY data whatsoever
- You can use it AND keep your data private
- We could, there’s code for ranking answers, or running models in parallel and voting
- No records? Privacy! GDPR is happy

Deployment as a service

Scientists from (currently just FZJ) can deploy their models on their hardware and point to blablador
This solves a bunch of headaches for researchers:
- Authentication
- Web server
- Firewall
- Availability
- Etc
If you have a model and want to deploy it, contact me!

What is it?

A classic MVC web application:
- Model: large language model(s)
- View: a web server and api server (openAI-compatible)
- Controller: coordinates the models
A collaboration with LM-Sys (From Vicuña 🦙 fame)
- FastChat: https://chat.lmsys.org
Python app, Runs on bare metal 🤘 with venvs
Models run in different sc_venv_templates
- Conflicting versions of libraries
Website Authentication: Helmholtz AAI, no records, just to keep the bots out
API Authentication: token with api scope from helmholtz codebase

OpenAI-compatible API

Uses openai-python from OpenAI itself
All services which can use OpenAI’s API can use Blablador’s API (VSCode’s Continue.dev, etc)
The API is not yet rate-limited, logged, monitored, documented or well-tested.

Jülich Supercomputers

Jülich Supercomputers

Well, no. Not really.

JSC CLOUD

Web server frontend
Authentication
API server
Workers’ controller

Haicluster

Currently hosting the workers (models)
2 nodes 4xNvidia 3090 24gb, 256gb RAM
1 node 8xNvidia 3090 24gb, 256gb RAM
GlusterFS:
- 6tb ssd for /home,
- 24tb hdd for data
Software: Ubuntu 20.04 LTS and EasyBuild

Demo: Website

Open https://helmholtz-blablador.fz-juelich.de
Play around! 🐶

Demo: API

Create a token in the helmholtz codebase settings with the api scope
Copy the token
Go to https://helmholtz-blablador.fz-juelich.de:8000/docs
Click on Authorize, and paste the token

Demo: API

Go to /v1/models
Click on Try it out
Click on Execute

Demo: API

Demo: cURL

curl --header "Authorization: Bearer MY_TOKEN_GOES_HERE"   https://helmholtz-blablador.fz-juelich.de:8000/v1/models

Demo: VScode + Continue.dev

Yes. It DOES run with Emacs too. Ask your favorite Emacs expert.
And vim as well. I guess.
But this demo is for VSCode. Sorry.
Add continue.dev extension to VSCode
On Continue, choose to add model, choose Other OpenAI-compatible API
Click in Open Config.json at the end

Demo: VScode + Continue.dev

Inside config.json, add at the "models" section:

    {
      "title": "Mistral helmholtz",
      "provider": "openai",
      "contextLength": 16384,
      "model": "Mistral-7B-Instruct-v0.2",
      "apiKey": "YOUR_TOKEN_GOES_HERE",
      "apiBase": "https://helmholtz-blablador.fz-juelich.de:8000"
    },

Try with the other models you got from the API!

Demo: VScode + Continue.dev

Select some code in a python file
Right click, Continue: Ask Code -> Add Highlited Code to Context
Ask Blablador to explain this code!
Can also fix, add tests, etc

Demo: VScode + Continue.dev

What can you do with it?

It’s being used in the wild!

https://indico.desy.de/event/38849/contributions/162118/

Dmitriy Kostunin’s talk earlier today at LIPS24

cosmosage is deployed on Blablador as of today!

It’s being used in the wild!

Someone reverse-engineered the API and created a python package

It’s being used in the wild!

https://git.geomar.de/everardo-gonzalez/blablador-python-bindings

It’s being used in the wild!

GEOMAR created a chatbot for their website
Scanned their material and created embeddings (RAG)
Calls Blablador’s API with embeddings and gets answers
Product of the NFDI hackathon DataXplorers // and beyond
It’s called TL;DR (Too Long; Didn’t Read) - Source code

It’s being used in the wild!

It’s being used in the wild!

EUDAT is a collection of data management services
Has an instance of NextCloud (File Share, Office, Calendar etc)
It’s integrating AI deeply into its services, backed by Blablador!

It’s being used in the wild!

Blassistant, a Blablador Nextcloud Assistant: https://github.com/EUDAT-B2DROP/assistant

It’s being used in the wild!

FZJ’s IEK7 (Stratosphere) is also using Blablador on their IEK7Cloud

Todo

Run gigantic models multi-node (eg. Falcon-180B)
Open for Helmholtz to have their models pointing to Blablador (no one asked yet)
Multi-GPU for SGLang (For Mixtral, and other models which are too big for my GPUs)
Multi-modal models (text+image, text+audio, etc)
Auto-RAG with privacy: don’t upload pdf, vector database in browser’s ram

Maybe

ARENA to evaluate models (that would imply keeping data)
There’s code for fine-tuning and training models, but no users
- Everyone chooses their own training model, this is just one more

Helmholtz BLABLADOR