Latent Space

Building Visual Intelligence — Andreas Blattmann & Dustin Podell, Black Forest Labs

8.5内容质量
Building Visual Intelligence — Andreas Blattmann & Dustin Podell, Black Forest Labs

TL;DR · AI 摘要

Black Forest Labs(BFL)通过FLUX 3模型推动视觉智能发展,结合多模态信号训练提升视频生成质量,并优化训练资源分配。

核心要点

  • FLUX 3模型通过同步训练音频、视觉和运动信号提升视频生成的物理合理性
  • 视频上采样器需保证时间一致性,避免帧间细节闪烁
  • 训练资源应优先投入能缩短推理时间的Diffusion模型优化

结构提纲

按章节快速跳转。

  1. 从Stability AI到BFL的技术路线迁移与FLUX系列模型发展

  2. FLUX 3通过同步处理视觉、音频和运动信号构建物理理解

  3. 视频上采样器需解决帧间细节不一致导致的视觉闪烁问题

  4. 对比Autoregressive与Diffusion模型的训练成本与推理效率差异

思维导图

用一张图看清主题之间的关系。

查看大纲文本(无障碍 / 无 JS 友好)
  • 视觉智能与FLUX 3
    • 多模态训练
      • 音频+视觉+运动信号同步
    • 视频生成技术
      • 时序一致性上采样器
    • 训练优化
      • Diffusion模型蒸馏
      • Autoregressive对比分析

金句 / Highlights

值得收藏与分享的关键句。

  • FLUX 3通过同步训练音频、视觉和运动信号,使模型能推断物体运动轨迹

    — 第3段

    ⬇︎ 下载 PNG𝕏 分享到 X
  • 独立上采样视频帧会导致细节闪烁,BFL视频上采样器通过时序一致性解决此问题

    — 第4段

    ⬇︎ 下载 PNG𝕏 分享到 X
  • Diffusion模型可通过蒸馏将采样步骤从数百步压缩至数步,显著提升推理效率

    — 第5段

    ⬇︎ 下载 PNG𝕏 分享到 X
#视觉智能#FLUX 3#Diffusion模型#Black Forest Labs
打开原文

Building Visual Intelligence — Andreas Blattmann & Dustin Podell, Black Forest Labs — Latent Space

This episode is sponsored by Deepgram. Try Deepgram for voice agents that truly understand your context! #contextiseverything

Play video

Open video on YouTube ↗

Not many people know that Stable Diffusion is why I went fulltime into AI (I aptly titled Multiverse, not Metaverse to call out Zuck’s mistaken investment). It turns out that Dustin Podell also had the same aha moment - quitting his day job when the model came out. But we took very different paths… I stuck to what I knew in human content creation, and Dustin immersed deep into AI content creation with Andreas Blattmann , co-inventor of Latent Diffusion and the rest of the world-leading imagegen crew at Stability AI. Stability fell apart, but Anj Midha helped the core group spin out to BFL, which immediately built an incredible new imagegen model in the Flux series. But why are they now going into video and world models?

Andreas gives an example you can try with one hand. If somebody tosses you a ball, your hand starts moving before the ball reaches you. You can do that without being able to write down the trajectory. When he talks about building visual intelligence , that’s the kind of knowledge he means. A still can show you where the ball is. A video shows you how it got there. The sound of impact can tell you when it hit something, even if the camera missed the contact. That’s BFL’s argument for training FLUX 3 on the three signals together. BFL’s FLUX 3 release

The horse in motion is one of the FLUX 3 examples we show in the intro. The audio examples are worth playing with the sound on. FLUX 3 Video

What a moving camera teaches you

Suppose the camera starts in front of a person and pans around them. At some point the model owes you the back of their head. Andreas thinks ordinary two-dimensional video contains a great deal of evidence about what objects look like from other angles. That doesn’t settle whether a model “understands physics,” a point we go back and forth on. It does give you a much better test than asking whether one frame looks convincing.

There’s a good engineering example late in the episode. Sharpen each frame of a clip independently and each frame might look terrific. Hit play and the extra detail can shimmer, because the upscaler made a slightly different decision every time. BFL derives its video upscaler from the same FLUX 3 backbone. Temporal consistency matters even when all the user asked for was a higher-resolution file.

What’s worth spending training compute on?

One of the questions I had for them was whether autoregression was taking over image generation. Andreas gives the other side its due. An autoregressive model can learn from all the tokens in a training example at once. On the diffusion side, the team can distill a long sampling process down to a handful of steps. Which advantage matters depends on whether you’re paying to train the model or waiting for an image to finish.

Dustin’s early Kontext experiment is a nice example of how they decide where to invest. He suspected an image model that could draw a dog wearing a hat already knew something about where hats go. He and a colleague collected a small set of photos labeled with relationships. A training run of a few hours got the model to put a logo on a shirt. It was rough, but it was enough to take to the next offsite. The team then spent months on the data and the model. FLUX Kontext examples

My favorite user story comes near the end. People were feeding FLUX old, degraded family photos, restoring them, then showing the pictures to their families. Dustin says he hadn’t predicted that use. After talking about preserving somebody’s identity across edits and across frames, it’s a lovely example of who actually notices if you get it right.

Sponsored

Your sponsor ad here

Reach the AI engineers building what comes next.

Latent Space is read by the people shipping agents, models and infrastructure. Enquire about sponsoring this slot.

[email protected] →

Show notes

  • BFL explains what a still image, a moving camera and a soundtrack each contribute during training. FLUX 3 launch and research overview
  • Watch and listen to the results of generating video and sound together. FLUX 3 Video demos
  • See how the lab teaches a model to make a requested change while preserving the rest of an image. FLUX Kontext examples
  • Andreas Blattmann co-created Stable Diffusion and co-invented the latent diffusion method behind it. Guest bio · Original research
  • Dustin Podell led FLUX Kontext and was the lead author of SDXL. Dustin’s project credits · FLUX Kontext · SDXL
  • The first Latent Space essay grew out of the Stable Diffusion release. Multiverse, not Metaverse

Chapters

0:00 Host introduction 3:39 Cold open 4:23 Black Forest roots 9:23 Dustin’s path 17:04 Visual intelligence 27:38 Inside SDXL 33:59 Learning motion 39:57 Flow matching 46:57 Starting BFL 50:36 Deepgram sponsor break 51:43 Early use cases for FLUX 54:58 Editing with Kontext 1:03:15 Training data 1:07:32 Testing an edit 1:13:12 Autoregression 1:20:50 World action models 1:30:08 Robotics in 2027 1:32:16 Who uses FLUX?