MOSS VL Realtime

Overview

MOSS-VL-Realtime is the realtime streaming checkpoint of the MOSS-VL release, part of the OpenMOSS ecosystem for open visual understanding.

Unlike offline video-language models that first read a complete video and then answer, MOSS-VL-Realtime is designed for continuous video streams. It perceives incoming frames and generates text in parallel, supports questions at arbitrary moments in the stream, and can decide whether to respond or keep observing when the visual evidence is insufficient.

This release keeps the MOSS-VL cross-attention design and a 256K text context window while adding realtime streaming data and an inference interface for timestamped frame-by-frame input.

Key Features

• Realtime streaming understanding: processes incoming frames continuously instead of waiting for a complete video.
• Interruptible interaction: users can ask questions at any timestamp in a running stream, and the model answers based on the frames observed so far.
• Proactive silence: the model can emit <|silence|> and continue observing when there is no meaningful visual update or the context is not sufficient.
• Dynamic correction: as new frames arrive, the model can revise earlier responses instead of being locked to an initial interpretation.
• Timestamp-aware frames: each streamed frame is associated with an absolute timestamp, helping the model reason about event order, duration, pacing, and fine-grained temporal localization.
• Unified MOSS-VL family: released together with MOSS-VL-Instruct and MOSS-VL-Base for offline use, continued pretraining, fine-tuning, and applied research.

Model Design

Architecture

MOSS-VL-Realtime adopts a cross-attention-based vision-language architecture that decouples visual encoding from language reasoning. This design is important for realtime usage because incoming visual content can be integrated into the running generation context without forcing the model into a strictly offline "load all frames, then answer" workflow.

Timestamp-aware Video Encoding

For video and realtime frame inputs, MOSS-VL injects absolute timestamps alongside sampled frames. This helps the model reason about when an event happens, how long it lasts, and how the scene changes over time instead of relying only on frame order.

MOSS-VL also uses Cross-attention Rotary Position Embedding (XRoPE), which maps text tokens and visual patches into a unified three-dimensional coordinate space defined by Time (t), Height (h), and Width (w). This gives the model a consistent positional representation for image, offline video, and realtime streaming video reasoning.

Configuration

Performance

MOSS-VL-Realtime is designed for streaming video understanding benchmarks where questions can arrive before a full video has been observed and correct answers may change as the scene evolves. It targets realtime interaction quality, proactive silence, and dynamic response updates in addition to standard video understanding accuracy.

Detailed benchmark tables and comparisons for this release will be maintained in the MOSS-VL project resources.

Quickstart

Installation

Clone the MOSS-VL repository and install the project requirements:

git clone https://github.com/OpenMOSS/MOSS-VL.git
cd MOSS-VL
conda create -n moss_vl python=3.12 pip -y
conda activate moss_vl
pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt

Load the Model

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

checkpoint = "OpenMOSS-Team/MOSS-VL-Realtime"

processor = AutoProcessor.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    frame_extract_num_threads=1,
)
model = AutoModelForCausalLM.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
model.eval()

If FlashAttention is unavailable in your environment, pass attn_implementation="eager" when loading the model.

Inference Examples

Online Inference

The recommended direct API is create_realtime_session(...). A service or application owns the video capture pipeline, converts camera, screen, or video-file input into PIL-compatible frames, and pushes each frame with a non-decreasing timestamp.

Common session operations:

• session.push_frame(image, timestamp=...) appends one visual frame.
• session.push_prompt("...") appends a user question while the stream is running.
• session.push_prompt_frame(prompt, image, timestamp=...) aligns a prompt with a specific frame.
• session.poll_output(...) or session.stream_outputs(...) returns incremental text chunks.

system_prompt and initial_prompt are tokenized as the initial system/user turns before the first frame arrives. Subsequent user turns can be appended with push_prompt(...) while the same session continues observing frames.

For complete real-time inference usage, including local-video replay and service deployment, see realtime_inference in the MOSS-VL GitHub repository.

import time
from PIL import Image

session = model.create_realtime_session(
    processor,
    initial_prompt=(
        "As the video streams frame by frame, describe important changes as they happen. "
        "Stay silent when there is no relevant update."
    ),
    frame_queue_size=256,
    max_tokens_per_turn=12,
    max_new_tokens=4096,
    do_sample=False,
)

frame_paths = [
    "data/frame_0001.jpg",
    "data/frame_0002.jpg",
    "data/frame_0003.jpg",
]

try:
    session.start()

    for index, frame_path in enumerate(frame_paths):
        image = Image.open(frame_path).convert("RGB")
        session.push_frame(image, timestamp=index / 1.0)

        while True:
            chunk = session.poll_output(timeout=0.0)
            if chunk is None:
                break
            print(chunk, end="", flush=True)

        time.sleep(1.0)

    session.push_prompt("What changed in the latest frames?")

    # Realtime sessions stay alive waiting for future input, so use a bounded
    # drain window and close the session explicitly when the producer is done.
    drain_deadline = time.monotonic() + 5.0
    while time.monotonic() < drain_deadline:
        chunk = session.poll_output(timeout=0.1)
        if chunk is not None:
            print(chunk, end="", flush=True)
finally:
    session.close()

Frame timestamps are measured in seconds and must be non-decreasing within a session. The input producer can be a camera, screen capture, decoded video file, browser frame sampler, or any other source that yields images with timestamps.

online_generate(...) is useful for backend systems that separate frame production and model inference through queues. It accepts dictionaries containing frames, prompts, events, reset controls, and stop controls.

import queue
import threading
from PIL import Image

input_queue = queue.Queue()
output_queue = queue.Queue()

worker = threading.Thread(
    target=model.online_generate,
    args=(processor, input_queue, output_queue),
    kwargs={
        "frame_queue_size": 256,
        "max_tokens_per_turn": 12,
        "max_new_tokens": 4096,
        "do_sample": False,
    },
    daemon=True,
)
worker.start()

input_queue.put({
    "initial_prompt": "Answer only when the streamed video provides enough evidence.",
})

input_queue.put({"frame": Image.open("data/frame_0001.jpg").convert("RGB"), "timestamp": 0.0})
input_queue.put({"frame": Image.open("data/frame_0002.jpg").convert("RGB"), "timestamp": 1.0})
input_queue.put({"prompt": "What is happening now?"})

try:
    while True:
        chunk = output_queue.get(timeout=0.5)
        print(chunk, end="", flush=True)
except queue.Empty:
    pass

input_queue.put({"stop_online_generate": True})
worker.join()

Each queue item can contain frame or image, timestamp, prompt, frames, event, events, initial_prompt, system_prompt, generate_kwargs, reset_session, or stop controls such as stop_online_generate.

Offline Inference

MOSS-VL-Realtime also keeps the offline helper APIs for image and video prompts. For purely offline use, MOSS-VL-Instruct is usually the preferred checkpoint, but the realtime checkpoint can still process complete image and video inputs.

video_path = "data/example_video.mp4"
prompt = "Describe this video."

text = model.offline_video_generate(
    processor,
    prompt=prompt,
    video=video_path,
    shortest_edge=4096,
    longest_edge=16777216,
    video_max_pixels=201326592,
    patch_size=16,
    temporal_patch_size=1,
    merge_size=2,
    video_fps=1.0,
    min_frames=1,
    max_frames=256,
    num_extract_threads=4,
    image_mean=[0.5, 0.5, 0.5],
    image_std=[0.5, 0.5, 0.5],
    max_new_tokens=256,
    temperature=1.0,
    top_k=50,
    top_p=1.0,
    repetition_penalty=1.0,
    do_sample=False,
    vision_chunked_length=64,
)

print(text)

offline_batch_generate accepts independent image/video/text queries. Queries in the same batch should share the same media_kwargs and generate_kwargs.

queries = [
    {
        "prompt": "Describe sample A.",
        "images": [],
        "videos": ["data/sample_a.mp4"],
        "media_kwargs": {
            "video_fps": 1.0,
            "min_frames": 8,
            "max_frames": 256,
        },
        "generate_kwargs": {
            "temperature": 1.0,
            "top_k": 50,
            "top_p": 1.0,
            "max_new_tokens": 256,
            "repetition_penalty": 1.0,
            "do_sample": False,
        },
    },
    {
        "prompt": "Describe sample B.",
        "images": [],
        "videos": ["data/sample_b.mp4"],
        "media_kwargs": {
            "video_fps": 1.0,
            "min_frames": 8,
            "max_frames": 256,
        },
        "generate_kwargs": {
            "temperature": 1.0,
            "top_k": 50,
            "top_p": 1.0,
            "max_new_tokens": 256,
            "repetition_penalty": 1.0,
            "do_sample": False,
        },
    },
]

with torch.no_grad():
    result = model.offline_batch_generate(
        processor,
        queries,
        vision_chunked_length=64,
    )

texts = [item["text"] for item in result["results"]]
print(texts)

Related Checkpoints

Limitations and Roadmap

MOSS-VL-Realtime is optimized for timestamped frame-by-frame streaming, but production latency depends on GPU hardware, frame sampling rate, transport overhead, and decoding speed. One model instance supports one active realtime session. The default frame queue bounds latency by dropping older pending frames when needed.

The model may emit realtime control tokens such as <|silence|>, <|round_start|>, and <|round_end|> depending on the application protocol. Downstream services should filter or render these tokens according to their UI needs.

We are continuing to improve realtime response timing, dynamic correction, broader streaming evaluations, RL post-training, and task-specific deployment recipes for future MOSS-VL releases.

Citation

@misc{moss_vl_2026,
  title         = {{MOSS-VL Technical Report}},
  author        = {OpenMOSS Team},
  year          = {2026},
  howpublished  = {\url{https://github.com/OpenMOSS/MOSS-VL}},
  note          = {GitHub repository}
}