> For the complete documentation index, see [llms.txt](https://langtech-bsc.gitbook.io/applied-intelligence/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://langtech-bsc.gitbook.io/applied-intelligence/using-the-models.md).

# Using the models

Access, runtimes and hardware for every recommended model.

## Getting access

All the models are on Hugging Face. Most download without any sign-up. **ALIA-40b-fc-2607** (and its GGUF version) is **gated**:

{% stepper %}
{% step %}

### Request access

Sign in to Hugging Face, open the [model page](https://huggingface.co/BSC-LT/ALIA-40b-fc-2607) and request access. Approval is manual.
{% endstep %}

{% step %}

### Authenticate your environment

Once approved, create an access token in your Hugging Face settings and make it available to your tools:

```bash
hf auth login            # or: export HF_TOKEN=<your-token>
```

{% endstep %}
{% endstepper %}

## Choosing a runtime

| Models                                                                                                                                              | Recommended runtime                                                 | Notes                                                                                                                                                      |
| --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [ALIA-40b-fc-2607](https://langtech-bsc.gitbook.io/applied-intelligence/models/text-models/alia-40b-fc.md)                                          | **vLLM** for full precision, **llama.cpp** for GGUF                 | Both serve an OpenAI-compatible API. The model card strongly discourages Ollama.                                                                           |
| [SalamandraTA](https://langtech-bsc.gitbook.io/applied-intelligence/models/machine-translation.md)                                                  | **vLLM** or Transformers for full precision, **llama.cpp** for GGUF | The model card strongly discourages Ollama. [mt-wrapper](https://github.com/langtech-bsc/mt-wrapper) adds Google Translate and DeepL-compatible endpoints. |
| [whisper-large-v3-ca-punctuated-3370h](https://langtech-bsc.gitbook.io/applied-intelligence/models/speech-models/whisper-large-v3-ca-punctuated.md) | **Transformers** (`pipeline`)                                       | Input audio at 16 kHz.                                                                                                                                     |
| [stt\_ca-es\_conformer\_transducer\_large](https://langtech-bsc.gitbook.io/applied-intelligence/models/speech-models/conformer-transducer-ca-es.md) | **NVIDIA NeMo**                                                     | Ships as a single `.nemo` file.                                                                                                                            |
| [Matxa-TTS v2](https://langtech-bsc.gitbook.io/applied-intelligence/models/speech-models.md)                                                        | BSC's **Matcha-TTS** fork, or the bundled **ONNX** model            | The ONNX file includes the vocoder, so it produces audio directly.                                                                                         |

## Serving LLMs with an OpenAI-compatible API

vLLM and llama.cpp's `llama-server` both expose the same `/v1` API as OpenAI. Your application code doesn't change when you move from a quantized build on a workstation to full precision on a GPU server: only `base_url` does.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="BSC-LT/salamandraTA-7b-instruct",
    messages=[{
        "role": "user",
        "content": "Translate the following text from Spanish into Catalan.\n"
                   "Spanish: El ayuntamiento abrirá el nuevo centro cultural el próximo lunes.\n"
                   "Catalan:",
    }],
    temperature=0.0,
)
print(response.choices[0].message.content)
```

Each model page has the exact command to start its server.

## Hardware

Approximate size of the weights. Leave extra memory for the context (KV cache), which grows with the input length.

| Model                                    | Format             | Weights        | Runs on                      |
| ---------------------------------------- | ------------------ | -------------- | ---------------------------- |
| ALIA-40b-fc-2607                         | BF16               | ≈ 81 GB        | 2 × 80 GB GPUs               |
| ALIA-40b-fc-2607                         | GGUF Q8\_0         | 43.0 GB        | 1 × 80 GB GPU                |
| ALIA-40b-fc-2607                         | GGUF Q4\_0         | 23.3 GB        | 1 × 48 GB GPU                |
| salamandraTA-7b-instruct                 | BF16               | ≈ 15.5 GB      | 1 × 24 GB GPU                |
| salamandraTA-7b-instruct                 | GGUF Q4\_K\_M      | 4.85 GB        | 8 GB GPU, or CPU             |
| salamandraTA-2b-instruct                 | BF16               | ≈ 4.5 GB       | 8 GB GPU                     |
| salamandraTA-2b-instruct                 | GGUF Q8\_0 / Q4\_1 | 2.40 / 1.52 GB | CPU                          |
| whisper-large-v3-ca-punctuated-3370h     | FP16               | ≈ 3.1 GB       | 8 GB GPU (CPU works, slowly) |
| stt\_ca-es\_conformer\_transducer\_large | NeMo               | 0.48 GB        | CPU or any GPU               |
| Matxa-TTS v2 (either model)              | PyTorch / ONNX     | 0.2–0.3 GB     | CPU or any GPU               |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://langtech-bsc.gitbook.io/applied-intelligence/using-the-models.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
