# Lance

URL: https://interfaze.ai/models/bytedance-researchlance

[All models](https://interfaze.ai/models)

Lance by bytedance-research, a any-to-any model with multimodal capabilities. Understand and compare multimodal features, benchmarks, and capabilities.

## Comparison

| Feature | Lance | Interfaze |
| --- | --- | --- |
| Input Modalities | text, image, video | image, text, audio, video, document |
| Native OCR | No | Yes |
| Long Document Processing | No | Yes |
| Language Support | unknown | 162+ |
| Native Speech-to-Text | No | Yes |
| Native Object Detection | No | Yes |
| Guardrail Controls | No | Yes |
| Context Input Size | 70K | 1M |
| Tool Calling | No | Tool calling supported + built in browser, code execution and web search |

### Scaling

| Feature | Lance | Interfaze |
| --- | --- | --- |
| Scaling | Self-hosted/Provider-hosted with quantization | Unlimited |

[Try Interfaze](https://interfaze.ai/dashboard)[Read the Docs](https://interfaze.ai/docs)

View model card on [Hugging Face](https://huggingface.co/bytedance-research/Lance)

## 🌟 Highlights

Lance is a lightweight native unified multimodal model that supports **image and video understanding, generation, and editing** within a single framework.

-   **Efficient at 3B scale.** With only **3B active parameters**, Lance delivers strong performance across image generation, image editing, and video generation benchmarks.
-   **Trained from scratch.** Lance is built with a staged multi-task recipe and trained entirely from scratch within a **128-A100-GPU** budget.

## 🎨 Demo

### Text-to-Video

### Video Editing

### Multi-turn Consistency Editing

### Intelligent Video Generation

### Video Understanding

### Text-to-Image Generation

### Image Editing

### Image Understanding

## 🚀 Installation

### Recommended Environment

-   **Software:** Python 3.10+, CUDA 12.4+ (required)
-   **Hardware:** A GPU with at least 40GB VRAM is required for inference

### Installation Steps

```
bash ./setup_env.sh
```

### Download Model Weights

Please download all the necessary model checkpoints of [Lance-3B (Huggingface Link)](https://huggingface.co/bytedance-research/Lance) and place them in the `downloads/` directory.

## 📚 Usage

### Inference

We provide a unified command-line interface for all generation / editing / understanding tasks:

#### Option 1: Configure and Run the Unified Script

```
bash inference_lance.sh
```

-   Before running, please configure the inference parameters at the top of `inference_lance.sh`.
-   **Supported tasks:** `t2i`, `t2v`, `image_edit`, `video_edit`, `x2t_image`, and `x2t_video`. You can modify `TASK_DEFAULT_CONFIGS` in `inference_lance.py` to customize the default data samples for each task.
-   **Note:** For all tasks, we recommend following the `prompt` format used in the provided examples when writing input prompts, as this typically leads to better generation quality.

#### Option 2: Configure and Run the Unified Script

We provide task-specific one-click commands for different generation, editing, and understanding tasks.

##### Text-to-Video Generation

```
bash inference_lance.sh \
  --TASK_NAME t2v \
  --MODEL_PATH downloads/Lance_3B_Video \
  --RESOLUTION video_480p \
  --NUM_FRAMES 121 \
  --VIDEO_HEIGHT 480 \
  --VIDEO_WIDTH 848 \
  --SAVE_PATH_GEN results/t2v_121f
```

##### Text-to-Image Generation

```
bash inference_lance.sh \
  --TASK_NAME t2i \
  --MODEL_PATH downloads/Lance_3B \
  --RESOLUTION image_768res \
  --VIDEO_HEIGHT 768 \
  --VIDEO_WIDTH 768 \
  --SAVE_PATH_GEN results/t2i
```

##### Video Editing

```
bash inference_lance.sh \
  --TASK_NAME video_edit \
  --MODEL_PATH downloads/Lance_3B_Video \
  --RESOLUTION video_480p \
  --SAVE_PATH_GEN results/video_edit
```

##### Image Editing

```
bash inference_lance.sh \
  --TASK_NAME image_edit \
  --MODEL_PATH downloads/Lance_3B \
  --RESOLUTION image_768res \
  --SAVE_PATH_GEN results/image_edit
```

##### Video Understanding

```
bash inference_lance.sh \
  --TASK_NAME x2t_video \
  --MODEL_PATH downloads/Lance_3B_Video \
  --RESOLUTION video_480p \
  --NUM_FRAMES 50 \
  --SAVE_PATH_GEN results/x2t_video
```

##### Image Understanding

```
bash inference_lance.sh \
  --TASK_NAME x2t_image \
  --MODEL_PATH downloads/Lance_3B \
  --RESOLUTION image_768res \
  --SAVE_PATH_GEN results/x2t_image
```

#### Available Tasks

| Task Name | Description | Example JSON |
| --- | --- | --- |
| t2v | Text-to-Video generation | config/examples/t2v\_example.json |
| t2i | Text-to-Image generation | config/examples/t2i\_example.json |
| image\_edit | Image editing | config/examples/image\_edit\_example.json |
| video\_edit | Video editing | config/examples/video\_edit\_example.json |
| x2t\_image | Image understanding | config/examples/x2t\_image\_example.json |
| x2t\_video | Video understanding | config/examples/x2t\_video\_example.json |

For understanding examples:

-   `config/examples/x2t_image_example.json`: image understanding examples for visual question answering and image-based reasoning.
-   `config/examples/x2t_video_example.json`: video understanding examples for video question answering and video captioning.

#### Parameters

You can configure the following hyperparameters at the top of the `inference_lance.sh` script:

| Parameter | Default Value | Description |
| --- | --- | --- |
| MODEL\_PATH | "downloads/lance\_3b" | Path to the downloaded Lance model weights. |
| NUM\_GPUS | 1 | Number of GPUs to use for inference. |
| VALIDATION\_NUM\_TIMESTEPS | 30 | Number of denoising steps (e.g., 30 or 50). |
| VALIDATION\_TIMESTEP\_SHIFT | 3.5 | Timestep shift parameter for flow matching scheduling. |
| CFG\_TEXT\_SCALE | 4.0 | Classifier-Free Guidance (CFG) scale for text conditioning. |
| VALIDATION\_DATA\_SEED | 42 | Random seed for generation reproducibility. |
| NUM\_FRAMES | 50 | Number of frames for video generation (Max: 121). Unused for image tasks. |
| VIDEO\_HEIGHT / VIDEO\_WIDTH | 768 | Spatial resolution. Unused for editing tasks (determined by input image/video). |
| RESOLUTION | "video\_480p" | Base resolution preset (image\_768res or video\_480p). |

### Gradio

```
python lance_gradio_t2v_v2t.py --gpus 0 --server-port 7860
```

### Benchmarks

#### DPG-Bench Evaluation

#### GenEval Evaluation

#### GEdit-Bench Evaluation

#### VBench Evaluation (Video Generation)

#### Running Benchmarks

Ready-to-run benchmark scripts are provided under `benchmarks/`:

| Benchmark | Modality | Script |
| --- | --- | --- |
| GenEVAL (image gen) | Image | benchmarks/image\_gen/GenEVAL/sample\_GenEVAL.sh |
| DPG (image gen) | Image | benchmarks/image\_gen/DPG/sample\_DPG.sh |
| GEdit (image edit) | Image | benchmarks/image\_gen/GEdit/sample\_GEdit.sh |
| VBench (video gen) | Video | benchmarks/video\_gen/Vbench/sample\_vbench.sh |

## 📄 License

Copyright 2025 Bytedance Ltd. and/or its affiliates.

## 💖 Citation

If you find **Lance** useful for your project or research, welcome to 🌟 this repo and cite our work using the following BibTeX:

```
@misc{lance2026,
  title  = {Lance: Unified Multimodal Modeling by Multi-Task Synergy},
  author = {Fengyi Fu and Mengqi Huang and Shaojin Wu and Yunsheng Jiang and Yufei Huo and Jianzhu Guo and Hao Li and Yinghang Song and Fei Ding and Qian He and Zheren Fu and Zhendong Mao and Yongdong Zhang},
  year   = {2026},
  note   = {Manuscript}
}
```

## 📞 Contact

For questions, issues, or collaborations, please contact [Mengqi Huang](https://corleone-huang.github.io/) and [Jianzhu Guo](https://guojianzhu.com/).

## Want more deterministic results?

[Try Interfaze](https://interfaze.ai/dashboard)[Read the Docs](https://interfaze.ai/docs)