视频模型的智能,大多来自语言

I have a pretty big claim the visual intelligence are actually mostly coming from language like these video models especially from now since the diffusion model technology is more mature like every time you see there there's some improvement on these models I would say mostly the the gain comes from language model not not coming from the the v the video model itself like the the video description models themselves.

我有一个相当大胆的论断:视觉智能其实大部分来自语言,比如这些视频模型,尤其是从现在开始,因为 diffusion model 技术已经更成熟了。每次你看到这些模型有什么进步,我会说绝大部分的提升来自语言模型,而不是来自视频模型本身,也就是不是来自视频描述模型本身。

Before we get into today's episode, I just have a small message for listeners.

在进入今天这一期之前,我想对听众们说几句话。

Thank you.

谢谢你们。

We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content.

如果你们没有选择点进来、收听我们的内容,我们就无法给你们带来这些你们如此明确地想要的 AI 工程、科学和娱乐内容。

We've been approached by sponsors on an almost daily basis.

几乎每天都有赞助商来找我们。

But fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way.

但幸运的是,你们当中有足够多的人真的订阅了我们,让这一切在没有广告的情况下也能持续下去,而我们也想一直保持这样。

But I just have one favor to ask all of you.

但我只想请求你们帮一个忙。

The single most powerful, completely free thing you can do is to click that subscribe button.

你们能做的、最有力也完全免费的一件事,就是点击那个订阅按钮。

It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring the Latent Space to you each and every week.

这是我唯一会向你们提出的请求,而它对我和我那个每周都拼尽全力为你们呈现 Latent Space 的团队来说,意味着一切。

If you do it, I promise you, we'll never stop working to make the show even better.

如果你们这么做了,我向你们保证,我们会永不停歇地把节目做得更好。

Now, let's get into it.

好,现在我们开始吧。

Okay, we're here in the studio with Ethan He, uh, most recently of xAI.

好,我们现在在录音室里,和 Ethan He 一起,呃,他最近是在 xAI。

Welcome.

欢迎。

Yeah, thank you.

好的,谢谢。

Glad being here.

很高兴来到这里。

We're also here with Vibhu.

跟我们一起在这儿的还有 Vibhu。

Uh you were first coming to us or joining Latent Space world because you were working on Cosmos and NVIDIA and you did a great paper.

呃,你最早来找我们、或者说加入 Latent Space 圈子,是因为你当时在做 Cosmos 和 NVIDIA 的工作,而且你还写了一篇很棒的论文。

We loved it.

我们非常喜欢它。

Uh you presented it as well.

呃,你还做了演讲。

So thank you for doing that.

所以谢谢你做的这些。

Yep.

对。

F also presented the moes.

F 也讲了 moe(混合专家)那部分。

Yes.

对。

Twice at Latent Space.

在 Latent Space 讲了两次。

Yeah.

是啊。

Yeah.

是啊。

Yeah.

是啊。

How did you actually hear about us?

你当初是怎么知道我们的?

Did we reach out to you?

是我们联系的你吗?

Is that how it worked?

是这么回事吗?

No, actually I the the community like I I realized oh there's this online community.

不是,其实是我……这个社区,我意识到,哦,有这么一个线上社区。

Yeah.

是的。

That people talk about AI and also learn learn from each other through papers every every week through the paper club.

大家在这里聊 AI,还通过每周的 paper club、通过论文互相学习。

It it's very nice.

这非常好。

I learned a lot

我学到了很多。

I think three years non-stop.

我想已经连续三年没停过了。

We haven't stopped even on Christmas and New Year's.

就算是圣诞节和新年我们也没停过。

Many weeks I want to stop.

有很多个星期我都想停下来。

No good.

不,挺好的。

I think you had posted that you worked on a paper and I was like oh very cool.

我记得你发过帖子说你做了一篇论文,我当时想,哦,太酷了。

We have paper club presented but I might have reached out to you after.

我们有 paper club 讲过它,不过可能是之后我又主动联系了你。

Yeah, because it's an amateur club, right?

对,因为它是个业余爱好者的俱乐部,对吧?

Uh so it's very unusual and but we have sometimes paper authors come by and and actually explain the paper.

呃,所以这挺不寻常的,不过我们有时候会请论文作者过来,亲自讲解那篇论文。

Today we just did uh the poolside paper which is apparently very good.

今天我们刚讲了呃 poolside 那篇论文,显然非常好。

Came out yesterday.

昨天才出的。

Uh pretty interesting, right?

挺有意思的,对吧?

Fully open.

完全开放。

They talk about everything system.

他们把整个系统的方方面面都讲了。

So it's a good one.

所以这是篇好文章。

We'll we'll recommend people to read it.

我们会推荐大家去读它。

Bring us up to speed on your transition to xAI because I actually don't even know when you joined.

跟我们说说你转去 xAI 的经过吧,因为我其实都不知道你是什么时候加入的。

uh just like tell us tell the story about the sort of transition

呃,就跟我们讲讲、讲讲这个转变的故事吧。

before xAI I was working on Cosmos world model as at NVIDIA.

来 xAI 之前,我在 NVIDIA 做 Cosmos world model。

So Cosmos is a the giant video foundation models that can that aims to simulate the world and for it serves as a foundation of for all of the roboticists to build on top of there.

Cosmos 是一个巨大的视频基础模型,目标是模拟整个世界,给所有做机器人的人当底座,让他们在上面继续搭建。

Once I built the Cosmos one, I realized as this thing also has a scaling law similar to language model.

把 Cosmos 做出来之后,我意识到这东西也有 scaling law,跟语言模型很像。

We need to scale scale up the video models further.

我们得把视频模型进一步 scale 上去。

Uh that's that's why I realized I need to move to somewhere with much more computer resources.

呃,正因为如此,我意识到自己得去一个算力资源多得多的地方。

That's how I

于是我就

then NVIDIA GPU came themselves.

然后 NVIDIA 的 GPU 自己就送上门了。

Yeah.

是的。

And timeline wise, when was Cosmo?

时间线上,Cosmos 是什么时候?

It was pretty early, right?

挺早的吧?

It was open world model paper.

就是那篇开源 world model 论文。

It was like uh end of 2024.

大概是呃 2024 年底。

End of 2024.

2024 年底。

Yeah.

对。

Then at at mid 2025 I moved to xAI.

然后 2025 年年中,我去了 xAI。

At that time I I joined by the time when XI was about to build video models and in multimodel models.

我加入的时候,正赶上 xAI 准备要做视频模型和多模态模型。

There were no no infra no data and no model.

当时没有 infra、没有数据、也没有模型。

And just as just a few engineers we we built it in three months and released the first model Grok imagine 0.9 and since then I I keep working on video models and move more from pre-training and to post training of the video models for example like reference to videos kind of like the cameo feature and the video extensions and now and uh before I left I I work time uh world model leading a small team to to focus on the real time long horizon video generation.

就靠几个工程师,我们花了三个月把它做出来,发布了第一个模型 Grok Imagine 0.9;从那以后我就一直做视频模型,并且从视频模型的 pre-training 更多地转向了 post-training,比如 reference-to-video 这种类似 cameo 的功能、还有视频续写;到最近,呃,离开之前,我在做 world model,带一个小团队专注做实时、长时程的视频生成。

Can you give like a rough road map of like okay you're on a brand new team Grock previously was only text or they partnered with BFL for uh their their image gen stuff what do you what are the building blocks right you have compute data you can procure somewhere like just you know what what are like the sequence of things that people should think about when you're setting up a new team I

你能不能大致讲讲那个路线图,比如说:你进了一个全新的团队,Grok 之前只有文本,或者说他们的图像生成是跟 BFL 合作的——那要搭建这么个东西,有哪些基础模块?你有算力,数据可以从某个地方采购,那……你懂的,当你要组建一个新团队时,大家应该按什么顺序来思考这些事?我

mean actually even deeper not just data you can procure you guys had to go through getting the data too right so

我是说,其实还要更深一层:不只是数据可以采购,你们还得亲自去把数据搞到手,对吧,所以

you shipped it pretty fast but yeah

你们上线得挺快的,不过好吧

yeah 3 months is like actually like very surprisingly fast.

对,三个月其实真的快得出乎意料。

Yeah.

对。

One thing I say like thanks to my experience at NVIDIA cuz first time when we were building Cosmos together, we built it uh for about a year.

有一点我得说,这要感谢我在 NVIDIA 的经历,因为我们第一次一起做 Cosmos 的时候,做了差不多一年。

So So this is like the second time I do it roughly roughly have an idea like what to do.

所以这是我第二次做,大致心里有数该怎么做。

I say the most important thing is is a talent.

我要说最重要的东西是人才。

Everyone everyone were very strong and clever very close with each other towards a common goal.

每个人都非常强、非常聪明,彼此关系很近,朝着一个共同目标使劲。

So that speed up things a lot.

这让速度快了很多。

So you reduce the communication bandwidth among people and everyone can can work toward the same goal.

这样就降低了人与人之间的沟通带宽,每个人都能朝同一个目标努力。

It's it's like every day there's not that much meetings on the calendar like maybe like a like a a sync a day and after that it's it's just all building.

那种感觉就是,每天日历上没那么多会,可能一天就一个 sync,开完之后就全是埋头做东西。

It was pretty fun at that time.

那段时间挺好玩的。

And another thing is that xAI has very strong foundations of like data data infer model infer and the the supporting there can can helps the model development a lot.

还有一点,xAI 在数据、data infra、model infra 以及配套支持上有非常强的底子,这些对模型开发帮助极大。

when I look at like training models, I don't uh so actually the the top important thing is like how many uh how many iterations can you do like per per day and and the the more iteration can you do you can you can train the model much faster.

我看训练模型的时候,呃,其实最最重要的一点是:你每天能做多少次迭代——你迭代次数越多,就能把模型训得越快。

So if you have very strong infra and you you have a lot of compute, you can you can train these models in very short period of time that can give you a much larger buffer to uh for errors and it also gives you the opportunity to spot more bugs.

所以如果你有非常强的 infra、又有大量算力,你就能在很短的时间里训出这些模型,这给了你大得多的容错缓冲,也让你有机会发现更多 bug。

Yeah.

嗯。

What is an iteration?

一次迭代具体指什么?

Is it like uh a few hundred steps or what?

是像呃几百步那样,还是别的?

Let's say just train training the model like from acquire new data and maybe design new algorithms and train a new model maybe a smaller scale.

就当作是训练模型吧,比如去获取新数据、可能设计新算法,然后训练一个新模型,规模可能小一些。

Yeah.

嗯。

So cycle time for like any hyperparam that you're

所以对于任何一个超参数,你的周期时间是

Yeah.

对。

cycle time end to end to like eval this model is this model better than my previous iteration.

端到端的周期时间,一直到评估这个模型、判断它是否比我上一次迭代更好。

Yeah.

嗯。

So like before you someone had already set this up that you can iterate very quickly.

所以说在你之前,已经有人把这套东西搭好了,让你可以非常快地迭代。

Yeah.

对。

I think the the foundation there is is extremely good for developing and research research models and often I find this is kind of boring but like a lot of the improvements does not come from new algorithms.

我觉得那里的基础设施对于开发和研究模型来说极其出色,而且我常常发现——这一点其实有点无聊——很多改进并不是来自新算法。

It comes from finding small bugs here and there in the data pipeline in the in the model training pipeline.

它们来自在数据管线里、在模型训练管线里到处找出一些小bug。

Zoske Z those gave the biggest boost to the model quality.

正是这些东西给模型质量带来了最大的提升。

It's interesting right?

挺有意思的,对吧?

So you say it's like small team less communication bandwidth but also a lot of quality is like fine little bugs.

所以你说这就像是小团队、更少的沟通带宽,但很多质量提升其实来自修掉那些小bug。

It seems counterintuitive right you have a lot of people you can iron out more of those but it's interesting to see the other side right.

这看起来有点反直觉,对吧,人多了你本可以排除掉更多这类问题,但能看到事情的另一面挺有意思的。

Yeah.

对。

Yeah.

对。

I also wonder have you do you try using LMS to look for bugs like I don't know I remember at that time it was mid 2025 so it's the the coding model wasn't quite there yet I remember I remember like December 2025 it was extremely good

我还好奇,你有没有试过用大模型来找bug?我不太确定,我记得那会儿是2025年年中,所以那时候的编程模型还没到那个水平;我记得到了2025年12月,它就变得极其好用了。

yeah I've been I've been using it at that time it's it's helpful sometimes it it produced codes that are kind of difficult to maintain even though like the first time it builds something extremely fast, but it gave the like a spaghetti code thousands of lines that I couldn't maintain and the OM itself couldn't figure out what's what's wrong and how to improve on top of it.

对,我那时候一直在用,它有时候很有帮助;它产出的代码有时候有点难维护,虽然它第一次能极快地搭出点东西,但它给的是那种上千行的意大利面条式代码,我没法维护,连大模型自己都搞不清哪里出了问题、以及该怎么在它基础上改进。

But now I find it much much much better.

但现在我觉得它好太太太多了。

Yeah, I want to bring up another point here is like now coding models are much more efficient and can help help us implement stuff much faster.

对,我想在这儿再提一个点,就是现在编程模型效率高多了,能帮我们更快地实现东西。

compute might become a bottleneck again because previously like if you want to train a new model say you want to generate new synthetic data and then or write a new algorithm it might take a few weeks and during that period of time you don't you might not have experiments to run but now you you can build that thing within a few hours then you can immediately train a model now you have to have enough compute to to try all of the ideas.

算力可能又会成为瓶颈,因为以前如果你想训练一个新模型,比如说你想生成新的合成数据,或者写一个新算法,可能要花好几周,而在那段时间里你可能没有实验可跑;但现在你可以在几个小时内就把那东西搭出来,然后马上就能训练一个模型,这时候你就得有足够的算力去尝试所有的想法。

So compute might be the bottleneck of iterating speed again.

所以算力可能又会成为迭代速度的瓶颈。

Yeah.

对。

Um I actually honestly I think it's like kind of a stressful job because you're like well I should be trying everything and if I'm not then I'm not doing my job well.

嗯,老实说我觉得这其实是个挺有压力的工作,因为你会想着我应该把每件事都试一遍,如果我没做到,那我就没把工作做好。

I mean there's also the stress of you're eating thousands of GPUs per hour which is very expensive and you know compute can go down the researchers

我是说,还有另一层压力:你每小时都在烧掉成千上万块GPU,这非常烧钱,而且你知道算力可能会掉线,研究员们就

daddy elong it was but you know there's still finite amount of compute like you want to use it you want to use it well you want more of it

老爹马斯克(会兜底)——不过你知道,算力终究是有限的,你想用它、想把它用好、还想要更多。

that was quite stressful indeed yeah I think one one thing is the with coding models now like a lot of these jobs can be automated which is much better.

那确实挺有压力的,对。我觉得有一点是,现在有了编程模型,很多这类工作都能自动化了,这就好多了。

Second, it's a it's a marathon, so you you got to maintain good health and uh the regular schedule.

第二,这是一场马拉松,所以你得保持良好的身体状态,还有呃规律的作息。

It's hard to hear that when you shift from zero to nothing in 3 months.

当你要在3个月里从零起步做出东西时,听到这话会觉得挺难做到的。

Yeah.

对。

[gasps] I mean, and I think obviously the the culture actually has very famously, you know, people people work very hard.

[倒吸一口气]我是说,而且我觉得很显然,那种文化其实非常出名,你懂的,大家都工作得非常拼。

Um, one thing I I I did want to dive into, you know, in our in the notes that you that you sent ahead of time, uh, you had specific comments about the cost of video gen training.

嗯,有一点我确实想深入聊聊,就是在你事先发给我们的笔记里,你对视频生成训练的成本有一些具体的评论。

Uh, presumably this is on like Colossus one, right?

呃,这大概是在Colossus 1上面吧,对吧?

Uh, the the 300 megawatt cluster and whatever you want to share.

呃,就是那个300兆瓦的集群,以及任何你愿意分享的内容。

I think there's there's three things you're talking about, right?

我觉得你说的其实是三样东西,对吧?

So there's videogen, there's also the image gen model that you put out.

有视频生成,还有你们发布的图像生成模型。

Do you want to like complete the Okay, so zero to one, you have a few months just what are the stages of create image?

你想不想把——好吧,那从零到一,你有几个月时间,创建图像都有哪些阶段?

Oh yeah, maybe I got distracted.

哦对,可能我刚跑题了。

Sorry.

抱歉。

Um and then you know from there there's video gen there's audio gen love to get into those things next but what is that first few months like so small team a lot of bugs iterations but like you know what does it look like do we take something off the shelf do we just get data compute what's what's the few months like how do you go to state-ofthe-art image gen model how do you just start

嗯,然后从那里往下,还有视频生成、音频生成,接下来很想聊聊这些;但最开始那几个月是什么样的?比如小团队、很多bug、不断迭代,但你懂的,它具体是什么样子——我们是拿现成的东西来用,还是就搞数据和算力?那几个月是什么样的?你是怎么做出一个state-of-the-art的图像生成模型的?你到底是怎么起步的?

echo comments specifically how I did but it's it's a quite standard process I can draw draw some examples from cosmos.

我具体讲讲我是怎么做的,不过这其实是个相当标准的流程,我可以拿 Cosmos 举几个例子。

So mainly it's like uh building building a video model you actually need to build a image model first and building building these two models the data you need is a 100% synthetic pair of language and image or language to video because on the on the internet actually the the videos don't naturally associate with text.

主要是这样,要训练一个视频模型,你其实得先训练一个图像模型,而要训练这两个模型,你需要的数据是 100% 合成的「语言-图像」或「语言-视频」配对,因为在互联网上,视频其实并不天然地和文本关联在一起。

So you can say, oh, like on YouTube, you have the title and you have the description and the comments of a video, but usually they're not relevant to to the video itself and say maybe like the video is a natural scene of mountains or something and the title is like I'm so happy today.

你可能会说,哦,在 YouTube 上视频有标题、有描述、有评论,但通常这些跟视频本身并不相关,比如视频拍的是山川自然景色之类的,标题却是「我今天好开心」。

So they have they have not no correlation at all.

所以它们之间根本没有任何关联。

So the first step is to you have to generate synthetic pair of language with the videos.

所以第一步,你得生成语言和视频的合成配对。

So you get the videos from the internet and you you use a VLM to caption the videos.

你从互联网上拿到视频,然后用一个 VLM 来给视频打字幕描述。

So that part here's a question like how do you how do you get the VLM to begin with?

那这里就有个问题:你一开始是怎么得到这个 VLM 的?

So like if there's no

就是说,如果一开始没有

you fuse the model right like

你把模型融合起来,对吧,就像

say if there's no like VLM exists like how do you generate the the text to the beginning right it's it's impossible

就是说,如果一开始根本不存在 VLM,你要怎么给最初的数据生成文本呢,对吧,这是不可能的

I see

我明白了

in the beginning it's like you ask human to describe the the video as as detailed as possible for example you ask them to describe everything like all objects all characters and all interaction and dialogues in the in the videos that In the protocol of cosmos labeling, we require the the objective we gave to the labelers was that you have to describe the video as detailed as possible such that a blind person shares a blob of text can reconstruct what the video is like from from their head.

一开始是这样的,你让人类尽可能详细地去描述视频,比如你让他们描述视频里的一切——所有物体、所有角色、所有互动和对话,在 Cosmos 标注的规范里,我们给标注员定的目标是:你必须把视频描述得足够详细,详细到一个盲人光凭这段文字,就能在脑海里重建出视频大概是什么样子。

Video or image.

视频还是图像?

You're talking about

你说的是

video or image.

视频或图像。

Either either one of them.

两者都可以。

This was pretty common when we went from like clip and dolly, right?

这在我们从 CLIP 到 DALL·E 那个阶段其实挺常见的,对吧?

It's all training on really detailed captioning of images.

全都是在对图像做非常详细的字幕描述来训练。

So same is applied to video but instead of using multimodal model to pass in video or images and write rich descriptions you can also

所以同样的做法也用在视频上,只不过除了用多模态模型输入视频或图像、生成丰富的描述之外,你还可以

I mean I think there's this traditional perspective of supervised uh or you know very highly human created thing.

我是说,我觉得有一种传统视角,就是靠监督学习,或者说很依赖人工创建的东西。

I feel like there's a unlock with unsupervised right where like you have enough to bootstrap that you can just throw common corpus on it or you know whatever uh like unsupervised vision and language pairing right like where you just have uh uh interspersed image and text and it just learns to me that is the VLM breakthrough that is different from the clip different from the the the prelim era

而我感觉无监督带来了一种解锁,对吧,就是你有足够的东西来完成冷启动,你可以直接把通用语料丢进去,或者随便什么,就像无监督的视觉-语言配对,你手里只有交错在一起的图像和文本,模型自己就学会了——对我来说这才是 VLM 的突破,这和 CLIP 不一样,和之前那个早期时代也不一样

yeah it's interesting to see that you kind of need both data Yes.

是啊,有意思的是可以看到你其实两种数据都需要,对。

For example, for you to bootstrap it up.

比如说,为了让它能冷启动起来。

Yeah.

对。

Yeah.

对。

For the generative model training, there's also usually like a small percentage of unlabelled data.

在生成式模型的训练里,通常也会有一小部分是没有标注的数据。

So, so the model is instructed to generate a video without any text instruction.

也就是说,模型被要求在没有任何文本指令的情况下生成一段视频。

So that that can also helps the model generalize.

这也能帮助模型更好地泛化。

So after after this stage of generation synthetic pair.

那么在这个生成合成配对的阶段之后,

So uh one uh one important common step is to train a compressor or a tokenizer of the image or videos.

呃,有一个重要的通用步骤,就是训练一个针对图像或视频的压缩器,或者叫 tokenizer。

So because uh if you train if you can technically theoretically train image or video models on pure pixels but the the problem is that the it's it's a lot of tokens.

因为呃,理论上你确实可以在纯像素上训练图像或视频模型,但问题在于 token 太多了。

So like one one image like uh it's a thousand by a thousand is like 1 million tokens 1 million pixels.

比如一张一千乘一千的图像,那就是 100 万个 token、100 万个像素。

It's impossible to train transformer on that.

在这上面训练 transformer 是不可能的。

So it's a you need to train a tokenizer which can go from image to latent space and latent space back to image.

所以你需要训练一个 tokenizer,它能把图像映射到 latent space,再从 latent space 映射回图像。

That's why we named the podcast.

这就是我们给这档播客起这个名字的原因。

Exactly.

没错。

But basically you're talking about vocabulary size.

不过基本上你说的就是词表大小(vocabulary size)。

So like what is what is like a million is impossible

所以说,一百万这个量级为什么是不可能的

in generative models.

在生成式模型里,

the the vocab is continuous.

这个词表是连续的。

It's a continuous space.

它是一个连续空间。

You can think about like you map an image to a vector.

你可以这样理解:你把一张图像映射成一个向量。

It's a it's a fixed length vector of like uh 16 or 48 something like that.

它是一个固定长度的向量,长度大概是呃 16 或者 48 之类的。

And then you you map that vector back to to the image space.

然后你再把这个向量映射回图像空间。

And the mapping is has the mapping is patchbased.

而这个映射是基于 patch 的。

So you say you have a 16x6 patch and you match you map that patch of pixels into this

比如说你有一个 16x16 的 patch,你把这块像素 patch 映射到这个

latent space.

latent space。

We've covered this like vision uh

我们之前讲过这个,就是视觉呃

yeah VA

对,VAE

you you basically compress your input you do your generation you're reasoning all that generation in smaller dimension and then you project back out.

你基本上是先把输入压缩,然后你在一个更小的维度里做生成、做所有那些推理运算,再把它投影还原回去。

Yeah,

对,

VA is a form compression, but I think the for me the patching thing is from VIT, right?

VAE 是一种压缩形式,但我觉得对我来说,patching 这个东西是来自 ViT 的,对吧?

You can make

你可以把

literally the Yeah, the the paper is titled like 16 by 16 is all you need.

字面上就是,对,那篇论文的标题就叫《16x16 就是你所需要的一切》。

Uh something like that.

呃,大概是这么个名字。

And then I think also people make a lot of comparisons with this kind of patching with convolutions.

然后我觉得大家也经常把这种 patching 和卷积做很多类比。

Yes.

是的。

Yes.

是的。

Which is you're you are kind of reconstructing the old paradigm with the new.

也就是说,你其实是在用新范式重新构建旧范式。

Yeah.

对。

Actually in VAE there there are both convolution networks and transformers.

其实在 VAE 里,卷积网络和 transformer 两者都有。

You you can actually do both.

你确实两种都可以用。

After this, so what you've got is you've got latent space tokens and you've got the the language tokens.

在这之后,你手上拿到的就是 latent space 的 token,以及语言的 token。

So now the training training of the diffusion transformer.

那么现在就到了训练 diffusion transformer 这一步。

Usually generative models use different transformers.

生成式模型通常用的都是 diffusion transformer。

It it's actually quite standard.

这其实相当标准。

It's it's very similar to how you train uh language transformer models.

它和你训练呃语言 transformer 模型非常相似。

There's not that much difference.

差别没那么大。

It's just the tokens the the visual tokens in visual tokens out.

只不过 token 换成了视觉 token,进去的是视觉 token,出来的也是视觉 token。

The only difference is there's a den noising process.

唯一的区别是多了一个去噪(denoising)过程。

So you train the model to unmask some of the noise.

所以你训练模型去揭开、去除一部分噪声。

So you you add you add random noise to the visual tokens and then you train the model to remove those noise to to generate a clean tokens.

也就是你往视觉 token 里加入随机噪声,然后训练模型把这些噪声去掉,生成干净的 token。

And in inference, the model can iteratively remove noise from 100% noise.

而在推理时,模型可以从 100% 的噪声出发,一步步迭代地去除噪声。

Yeah.

对。

And then there's also uh to speed things along on the tech tech tree of diffusion, there's cfg like uh and then there's also I guess latent diffusion that you know there's someone in there.

然后为了在 diffusion 这棵技术树上加快进度,还有呃 CFG 之类的东西,另外我猜还有 latent diffusion,你知道,里面涉及到某些人的工作。

I think somewhere along the line obviously like stability and all these other guys pioneered a lot of this like um architecture.

我觉得在这条发展线上的某个节点,显然像 Stability 这些人开创了很多这类呃架构。

I don't know if you want to get into that or just or do the the video side up to you.

我不知道你是想深入聊这个,还是就直接讲视频那边,由你来定。

After you train such model such image model, the reason it's a it's a foundation for video models is that image image models are cheaper to train and they have much denser connection between language and text.

当你训练好这样一个模型、这样一个图像模型之后，它之所以能成为视频模型的基础，是因为图像模型训练成本更低，而且它们在语言和文本之间有着密集得多的连接。

So uh sorry, language and images.

呃，抱歉，是语言和图像之间。

For example, you you train a billion you turn on a billion images and there's a mapping from from the tax to to the image and the cost to train the same like the a billion a billion tax to a billion videos that that's much more expensive because videos naturally have more tokens than images because the diffusion models their understanding of language purely come from this this mapping.

举个例子，你训练十亿张图像，从文本到图像有一套映射；而如果要用同样的量、比如十亿条文本对十亿个视频去训练，那成本就高得多，因为视频天然比图像有更多的 token——diffusion 模型对语言的理解完全来自这套映射。

So if you don't have enough mapping, so if you only train like a 10 million videos or something there, you might not see enough language tokens in your training.

所以如果你没有足够的映射，比如你只训练了大约一千万个视频之类的，那你在训练里可能看不到足够多的语言 token。

So your model does not understand human intention enough.

这样一来你的模型就无法充分理解人类意图。

So that's why you usually you train you first train this image diffusion models and then you bootstrap the video model from there.

所以这就是为什么通常你会先训练这种图像 diffusion 模型，然后再从这里把视频模型 bootstrap 出来。

One thing I did want to ask because I actually I think you're you're the first video model person I've ever talked to.

有件事我确实想问，因为我觉得你可能是我聊过的第一个做视频模型的人。

I think we we've we've like talked to Luma and and all those folks.

我们其实跟 Luma 以及那些人都聊过。

There's all these tricks in video compression where basically frame by frame there's not that much difference.

视频压缩里有各种技巧，基本上逐帧来看差异并没有那么大。

So actually you don't have to regenerate or resave the whole frame, right?

所以其实你不必重新生成或者重新保存整帧，对吧？

Um I think MP4 compression or something else like that.

嗯我在想 MP4 压缩或者类似的其他方法。

Uh is it tempting to use that or as far as I can tell everyone just treats it as no we will just generate every frame.

呃用那种方法有诱惑力吗？还是说据我所知大家都只是把它当成——不，我们就是要生成每一帧。

Is that roughly the state of the art?

这大致就是当前的技术前沿状态吗？

There are a few different approaches.

有几种不同的路子。

Let's say first like you you want to just directly use MP4 compression and you use that as a tokens for the transformers to train right.

先说第一种，假设你想直接用 MP4 压缩，把它作为 token 喂给 transformer 去训练，对吧。

So people actually have tried that but the the main challenge is the latent space for the MP4 tokens were not were not very comprehensible for the models.

人们其实试过这个，但主要的挑战在于 MP4 token 的 latent space 对模型来说不太可理解。

It's it's extremely hard to train on that and there's a so that's why we created ves which creates more continuous uh latent space so the models can understand that latent space and learn from it much easier even within the ves there are different difficulties of the Latent Space so you you can imagine something the the simplest the most naive way is like you have an image and you just shuffle all of the images into a into a vector.

在那上面训练极其困难，所以这就是为什么我们创造了 VAE，它能产生更连续的 latent space，让模型能更容易地理解这个 latent space 并从中学习；即便在 VAE 内部，latent space 也有不同的难度等级，你可以想象——最简单、最朴素的做法就是，你有一张图像，直接把整张图像打散成一个向量。

So you don't need to train any ve right but that space is extremely hard for models to train on top of that that's why there's some debate on like how do you compress the the tokens.

这样你就不需要训练任何 VAE，对吧，但那个空间对模型来说极其难以在其上训练，这就是为什么关于如何压缩 token 存在一些争论。

So so you mentioned like you can compress frame by frame also you can compress uh the temporal dimension.

所以你刚才提到，你可以逐帧压缩，也可以在时间维度上压缩。

Yes.

对。

The difference is if you compress temporal dimension you you get a much higher compression rate is because there there's temporal redundancy between frames because this frame and the last frame likely they are mostly similar.

区别在于，如果你在时间维度上压缩，你能得到高得多的压缩率，这是因为帧与帧之间存在时间冗余——因为这一帧和上一帧很可能大部分是相似的。

So there's only some small difference.

所以只有一些小差异。

Uh for example like uh I think in one 2.1 they have like a 8 by 8x4 compression rate.

呃比如说，我想 Wan 2.1 里他们用的是 8x8x4 的压缩率之类的。

So the the four temporal tokens are compressed into into one tokens that can save a lot of save save a lot of the context length.

所以那四个时间维度的 token 被压缩成一个 token，这能省下很多、省下很多上下文长度。

If you do it frame by frame you have to do maybe like 8 by 8 by one your context lens will be four times larger.

如果你逐帧来做，你可能得做成 8x8x1，你的上下文长度就会大四倍。

That being said, the benefit of the frame per frame compression, we might come back to this later, is uh real timeness and interactivity because if you if you strain the output of the model uh frame by frame, you can the model can respond to any user requests immediately.

话虽如此，逐帧压缩的好处——我们可能稍后会回到这个话题——是实时性和交互性，因为如果你逐帧地流式输出模型的结果，模型就能对任何用户请求立即做出响应。

So if you have like a temporal four uh four compression four times compression then

所以如果你用的是时间维度四倍、呃四倍压缩、四倍压缩，那么

it might be laggy.

它可能就会卡顿。

Yeah there there's a lag there in nature.

是的，那里天然就有延迟。

So you're very pickled on this.

所以你在这一点上很执着。

Uh let's just go ahead and bring it up cuz we have to visual anyway.

呃我们干脆直接把它调出来吧，反正我们也需要点画面。

There's some frontier applications of real time video gen.

实时视频生成有一些前沿的应用。

So flipbook is one of the examples that went viral recently.

flipbook 就是最近爆火的例子之一。

Right.

对吧。

What is flip book?

flipbook 是什么？

Flipbook is is kind of like a web brow web browser.

flipbook 有点像一个网页浏览器。

You can see like it has the web browser UI on top.

你能看到它顶部有网页浏览器的 UI。

The difference is all of the UIs are generated by generative image model in real time and anything here are are fake.

区别在于，所有这些 UI 都是由生成式图像模型实时生成的，这里的一切都是假的。

But you can you can explore inside this imaginary world.

但你可以在这个想象出来的世界里探索。

Say like we here we have engineering the grid pyramid like the the model generated this for us to understand how it works and if we want to navigate around and understand further we can click on some of the some of the description here and the model will generate a new page new sub page describing the details we want to know about.

比如我们这里有工程学的金字塔网格——模型为我们生成了这个，好让我们理解它是怎么运作的；如果我们想四处浏览、进一步了解，我们可以点击这里的某些描述，模型就会生成一个新页面、一个新的子页面，来描述我们想知道的细节。

So it's basically kind of we are playing a video uh but it's pausing for our next interaction and then it just plays the next thing based on our interaction.

所以它基本上有点像我们在播放一段视频、呃但它会停下来等我们的下一步交互，然后根据我们的交互再播放接下来的内容。

Which is kind of cool.

这挺酷的。

Yes.

是的。

Yeah.

对。

You kind of decide your story.

你算是自己决定你的故事走向。

So this was you know how do you make a pyramid uh levering technique seemed interesting right?

所以这个就是——你知道该怎么造一座金字塔，呃用杠杆技术看起来挺有意思的，对吧？

It shows how do you take Okay.

它展示了你该怎么把——好。

I want to know

我想知道

the demo the demo tweet had more animation between frames.

那条 demo 推文里帧与帧之间有更多的动画。

I think it's just skipping uh

我觉得这里只是跳过了呃

just skipping a lot of frames.

只是跳过了很多帧。

They also have a video mode but uh I guess a lot of people are using it.

他们也有一个视频模式，不过呃我猜很多人都在用它。

There's a live video stream.

有一个实时视频流。

We can try um

我们可以试试嗯

Yeah.

是的。

So, so this is an example of the kind of future that you see at the extreme.

所以，所以这就是一个例子，展示了你在极端情况下看到的那种未来。

We don't we're obviously not in it today but in a world where inference is completely free.

我们显然今天还没到那一步，但在一个推理成本完全免费的世界里。

This is better than generating code in text.

这比用文本生成代码要好。

Yeah.

对。

So this is this is the final state of VR viva be at for world model I think imagine imagine internet doesn't exist and then you type in google.com like what should what should a model show you um the model can imagine something and this is what the model imagine and these web pages they completely do not exist so I think as a inference cost come down we are going to have generative UI for everything.

所以这就是 world model 的终极形态,我觉得——想象一下互联网根本不存在,然后你输入 google.com,那模型应该给你显示什么?嗯,模型可以想象出某种东西,而这就是模型想象出来的,这些网页是完全不存在的。所以我觉得随着 inference 成本下降,我们会为一切都用上生成式 UI(generative UI)。

If you think about how the coding model works so they they write code for a web page and they render the code might be converted into binary and the binary render the pixels on the screen.

你想想 coding 模型是怎么工作的:它们为一个网页写代码,代码渲染时可能会被编译成二进制,再由二进制把像素渲染到屏幕上。

So we in machine learning every time we have some breakthrough obviously it's more inent.

所以在机器学习里,每次我们有突破,显然它都更端到端(more inherent,即更直接)。

So why don't we have like a user instruction to the pixel directly?

那我们为什么不能直接从用户指令跳到像素呢?

So the generated UI will be user intention to to the pixels directly and say like even if I want email let's say everyone everyone have the same interface but I want I want it slightly different.

所以生成式 UI 就是把用户意图直接映射到像素上——比如说,哪怕我要收邮件,假设每个人用的都是同一个界面,但我想要稍微不一样的。

I want the email to show show to me like a Tik Tok so I can swipe left and right for the emails and or maybe you want something else.

我想让邮件像 TikTok 那样呈现给我,这样我就能左右滑动来看邮件,或者你也可能想要别的什么。

we can have complete different things or like I have I'm looking at Instagram stories.

我们可以有完全不同的东西,再比如我在看 Instagram stories。

I don't like the like button.

我不喜欢那个 like 按钮。

I always make click it and gener the UI without it.

我老是不小心点到它,那就生成一个没有它的 UI。

So it's going to be a revolutionary replacement of the interface.

所以这将是对界面的一次革命性替代。

So in the future we might have much more powerful OM and coding models running behind the scene and in the in the front end the diffusion model will actually be the front end to show stuff to you.

所以未来我们背后可能跑着强大得多的大模型和 coding 模型,而在前端,diffusion 模型实际上会成为那个把内容展示给你的前端。

That's how I imagine it.

我大概就是这么想象它的。

Yeah, diffusion front end deterministic back end.

对,diffusion 做前端,确定性(deterministic)的后端。

Yes,

对,

something like that.

差不多就是那样。

I find it very expensive but uh you know

我觉得这非常烧钱,不过呃你懂的

I find it interesting you called LM's writing code on the back end deterministic but okay.

我觉得挺有意思的,你把在后端写代码的大模型称为确定性的,不过好吧。

Yeah, you write it once compared and then you execute.

对,你写一次、编译好,然后就执行。

If you think about the cost, say let's say H100 cost $1 per hour and if you use this 8 hours a day and 30 days.

你算算成本,假设一块 H100 每小时 $1,你每天用 8 小时、用 30 天。

So um every month you're paying this 240.

所以嗯每个月你要付这 240。

You likely not not want to pay for that.

你多半不愿意为此付钱。

That's even more expensive than cloud code max.

那甚至比 Claude Code Max 还贵。

But if you think about the the compute costs come down like two times every year, I think that the future will likely arrive,

但你想想算力成本每年会下降差不多两倍,我觉得那个未来很可能会到来,

right?

对吧?

Compute cost comes down, compute gets faster, model gets smarter, model gets smaller.

算力成本下降,算力变快,模型变聪明,模型变小。

Yeah, I don't know why you say two times because I think it's like 100 times.

对,我不知道你为什么说两倍,因为我觉得是 100 倍那种级别。

In language models, it is roughly 100 to a,000 times every 12 to 18 months uh for the same given level of LMS uh ELO

在语言模型里,对于同一个给定水平的大模型 ELO,大概每 12 到 18 个月就是 100 到 1,000 倍。

that that's a net of everything, right?

那是把所有因素都算进去后的净效果,对吧?

That's model performance alongside compute.

那是模型性能和算力一起在提升。

So different than just compute cost come down, but um you know a very interesting future.

所以跟单纯的算力成本下降不一样,不过嗯,是个非常有意思的未来。

Yeah.

对。

Um

嗯

for the the web designers will have to shout out that accessibility is an issue, right?

对做网页设计的人来说,得提一句无障碍(accessibility)是个问题,对吧?

Like you know how do you deal with screen readers or whatever?

比如你知道要怎么处理屏幕阅读器之类的东西?

But yes, this is higher bandwidth storytelling than anything you can possibly generate with code.

不过没错,这种叙事的信息带宽比你用代码能生成的任何东西都高。

Right?

对吧?

So I think that's the the rough idea.

所以我觉得大致就是这个思路。

I'd like to add a little bit that so human naturally have the maximum bandwidth when we are looking at things look at videos and and we also have maximum output bandwidth when we are talking.

我想稍微补充一点,人类天生在看东西、看视频的时候输入带宽最大,而我们在说话的时候输出带宽最大。

So in the future it might be something like we we talk to AI models and the AI model responds back with the generative UI.

所以未来可能会变成这样:我们对 AI 模型说话,AI 模型用生成式 UI 回应我们。

So that that will be the maximum input and output bandwidth to interact with AI models before neural link happens.

所以在 neural link(脑机接口)出现之前,这将是与 AI 模型交互的最大输入和输出带宽。

And I mean it's also very custom right some people are very visual some people are not as visual right they prefer the text but the best thing about generative UI right can also be text.

而且我是说,这也非常因人而异,对吧,有些人非常视觉化,有些人不那么视觉化,他们更喜欢文字,但生成式 UI 最棒的一点是,它也可以是文字。

Yes,

对,

there's another project that we wanted to highlight which is the Neuro OS.

还有一个我们想重点讲一下的项目,就是 Neuro OS。

Kind of similar idea, but here you're literally operating uh uh simulating an operating system with a video model.

思路有点类似,但在这里你真的是在用一个视频模型来运行——呃呃——模拟一个操作系统。

Um and you can play Doom, you can Firefox.

嗯,你可以玩 Doom,可以用 Firefox。

Uh I find this like mildly less impressive obviously because it's an OS that I can run.

呃我觉得这个稍微没那么惊艳,显然是因为操作系统我自己就能跑。

But here everything is imagined.

但在这里,一切都是想象出来的。

I I was you know used to the command W to close the Firefox tab.

我你知道吗,习惯了用 Command W 来关 Firefox 标签页。

I didn't crash.

我没崩溃。

It's too immersive.

它太沉浸了。

It's too immersive for me.

对我来说它太沉浸了。

I wanted to close the tab.

我想关掉那个标签页。

But yes, I can play generated diffuse.

不过没错,我能玩那个生成的 diffuse(扩散画面)。

This is shockingly fast.

这快得惊人。

Yeah,

对,

because I I remember there was a demo about like maybe one or two years ago.

因为我记得大概一两年前有过一个 demo。

Someone tried to do the first person shooter with a image model.

有人试着用一个图像模型做第一人称射击游戏。

There was no consistency.

完全没有一致性。

It was very slow.

非常慢。

But here like realistically it's this is Doom.

但在这里,真的就很逼真了,这就是 Doom。

I mean I think there's two sides to that, right?

我是说我觉得这有两面,对吧?

There's like okay, what is running a game?

就像,好,运行一个游戏是怎么回事?

The heavy part of it is actually the game engine, all the lighting, all that stuff, the graphics.

它最吃重的部分其实是游戏引擎、所有光照、所有那些东西、图形。

This is just kind of video, right?

而这个只是某种视频,对吧?

Like we've solved consistency.

比如我们已经解决了一致性。

This is still, you know, it looks like a few years old image generation.

这个嘛,你懂的,它看起来像是几年前的图像生成水平。

There's some temporal consistency, but it's it's kind of just images stitched together as frame video.

有一些时间上的一致性,但它基本上就是把一堆图像拼成逐帧视频。

But it it's a good visual representation to p to picture the future you want to see, right?

不过它是一个很好的视觉呈现,用来描绘你想看到的那个未来,对吧?

Like that's that's what I see in these.

就像,这就是我在这些东西里看到的。

more.

更多。

So,

所以,

this reminds me of how how the video models gets better and better.

这让我想起视频模型是如何越来越好的。

So, neuro OS is kind of if you just look at it, it feels like it's just a a a crappy version of the like the the Windows we could have, right?

所以 neuro OS,如果你光看它,会觉得它只是我们本可以拥有的那种 Windows 的一个很糙的版本,对吧?

And but but the difference is so the model this model is overfitted on the the existing operating systems.

但区别在于,这个模型是在现有的操作系统上过拟合(overfitted)出来的。

it it can generate nothing different than that but it's actually also similar to video models.

它生成不出跟那不一样的东西,但这其实也和视频模型很像。

So when we're training these video model image model we train them on internet there's no imaginary supernatural stuff on the internet but once we train this model you can prompt the model to generate something supernatural that have never existed in the data set.

所以我们在训练这些视频模型、图像模型时,是拿互联网数据训练的,互联网上没有想象中的超自然的东西,但一旦训练好这个模型,你就可以提示它生成某种超自然、在数据集里从未存在过的东西。

So if you train your neuro OS or neuro computer on the standard screen recordings on the entire internet the model can imagine completely new interface to to interact with a computer.

所以如果你拿整个互联网上的标准屏幕录制来训练你的 neuro OS 或 neuro computer,这个模型就能想象出全新的、跟计算机交互的界面。

Yeah, this is one of those things that is magical to me.

对,这是那种对我来说很神奇的事情之一。

Uh usually generalizing auto distribution is bad but somehow we have learned some kind of internal world model.

呃通常泛化到分布外(out-of-distribution)效果是很差的,但不知怎么的,我们已经学到了某种内部的 world model。

Yes,

对,

that you say, you know, this plus but it looks like rainbows and butterflies.

就是你说,你知道的,这个加上但它看起来像彩虹和蝴蝶。

It'll do it and it'll kind of make sense.

它就真会那么做,而且还显得挺说得通的。

Yeah.

对。

So, yeah, that's kind of cool.

所以,对,那挺酷的。

Yeah, I I don't know if there's any comment more on there.

对,我不知道在那上面还有没有更多可说的。

I I do I do wanted to I did wanted to touch a little bit more on the model architecture stuff which I think you were getting it's like really fascinating.

我确实,我确实想再多聊一点模型架构方面的东西,我觉得你之前正要讲到,那真的很迷人。

We we don't get a chance to talk about this enough.

我们没什么机会把这个聊透。

So one of the papers that we covered we've covered every annual uh segment anything release uh and I don't know if you follow I mean you're a computer vision guy so you you know so they did memory attention which is kind of interesting and I always think like anything where you can across the temporal dimension keep some consistency um I think it's like very fascinating and I don't know if basically like does that the CV side bleeding into videogen side I think is underexplored right like we talked about it for labeling but you can borrow the architecture itself.

所以我们讲过的一篇论文——我们把每一次 Segment Anything 的年度发布都覆盖过了——呃我不知道你有没有关注,我是说你是搞计算机视觉的,你懂的,所以他们做了 memory attention,这挺有意思的,我一直觉得任何能跨时间维度保持某种一致性的东西,呃我觉得都非常迷人,而且我不知道,基本上就是说,CV 这边渗透进 videogen 那边,我觉得是被探索得不够的,对吧,就像我们聊过它用在打标(labeling)上,但你其实可以把架构本身借过来。

And there's there's also complete different approaches, right?

而且其实还有完全不同的路线，对吧？

Like you you brought up the term world model.

比如你刚提到了 world model 这个词。

So we went from video model to world model.

所以我们从 video model 一路聊到了 world model。

There is diffusion, but there's also other approaches that people are doing.

有 diffusion，但大家也在做其他的路线。

So maybe we get into those after as well, you know.

所以也许我们之后也可以聊聊那些，你懂的。

Yeah, he has a whole definition of world models and stuff.

对，他对 world model 之类的有一整套自己的定义。

I I feel like we threw a lot at you.

我感觉我们一下抛给你太多东西了。

Whatever you want to comment on.

你想评论哪个都行。

I I think one thing that we should actually comment back on is like okay so we were talking about the steps to train image gen to video model one thing we don't see as much of is like okay you brought up the delta in training data right so you won't have as much a video model might not generalize but what is the cost of training a large video model so we know for LLM's roughly okay uh even like the poolside thing that came out today right it's a gemma level model trained on roughly 40 trillion tokens at this many H200's over this much time, right?

我觉得有一点我们其实应该回过头来讲一下，就是好，我们刚才在聊从 image gen 训练到 video model 的步骤，有一点我们平时看得不多，就是——好，你提到了训练数据上的差异，对吧，所以你不会有那么多数据，一个 video model 可能泛化得没那么好，但训练一个大型 video model 的成本到底是多少？我们对 LLM 大概是知道的，好，呃，就连今天出的那个 poolside 的东西，对吧，它是一个 gemma 级别的模型，用大约 40 万亿 token、在这么多张 H200 上训了这么长时间，对吧？

You can see what is the exact cost of that.

你能看到它确切的成本是多少。

So how many GPU hours over how much H200 cost?

也就是多少 GPU 小时、乘以多少 H200 的成本？

So how how do we do the backend math of you know same thing for video models, image models.

那我们怎么来算 video model、image model 这一块背后的这笔账呢？

How do you how do you kind of break that down?

你会怎么去拆解这个成本？

I can share some back analog calculation.

我可以分享一些粗略的类比估算。

So surprisingly video models is like the the cost is very is comparable to language models and I'll be the largest scale as language model maybe like a medium scale language models.

说来意外，video model 的成本其实和语言模型很接近，我会说相当于最大规模的语言模型，或者说相当于一个中等规模的语言模型。

I said just storing the videos alone it it costs a lot.

我说光是存储这些视频本身就要花不少钱。

You can you can maybe look up on AWS or something

你可以去 AWS 之类的地方查一下。

usually like say if you have a billion videos and let's say let's just say like each video like five megabyte then you need like five pyabyte to just store those videos and also remember we talked about you use a VE to compress the videos and you also need to store typically you need to store those continuous feature and also in your storage that's also comparable size with the videos themselves.

通常来说，比如你有十亿个视频，假设每个视频大概 5 兆，那你就需要大约 5 PB 来单纯存这些视频，而且别忘了我们聊过你要用 VAE 去压缩视频，你还得存那些连续的 feature，在存储里这部分的体量通常也和视频本身相当。

So just storing storing these videos and the features is it's tens of pabytes alone and uh

所以光是存这些视频和 feature，就要几十 PB，而且呃……

I just I just looked up the calculation 5 pabytes on S3 standard is 100k per month.

我刚查了一下，5 PB 存在 S3 标准存储上是每月 10 万美元。

Okay.

好。

And you need

而且你还需要——

and then like a tens 200k and even more expensive is you have the ingress and egress like through the internet.

然后大概是几十万、二十万美元，更贵的是你还有通过网络的 ingress 和 egress。

You have to just to download those videos.

你光是要把那些视频下载下来。

I believe it's it's more expensive on AWS than just storing those videos

我相信在 AWS 上这比单纯存那些视频还要贵。

and each training runs you probably need to pull them once.

而且每一轮训练你大概都得把它们拉取一次。

If you train multiple times it's it's even more than that.

如果你训练多次，那就比这还要多。

So, so it's like just storing storing on the network those cost is is just I guess it would be a few a few millions per months to just storing everything not to mention the GPU cost.

所以，光是把这些东西存在网络上，这笔成本，我估计单是把所有东西存下来就要每月好几百万美元，更别提 GPU 的成本了。

Yeah,

对。

my my side tangent like you know the compute rental like GPU rentals very efficient.

我插一句题外话，就是你知道，算力租赁——GPU 租赁——效率很高。

There's one side okay you can be xAI and build your data center should we not just build our like storage compute as well like cloud cost compared to just you know yeah exactly especially with like egress and stuff so you know

有一种思路是，好，你可以像 xAI 那样自建数据中心，那我们是不是也该把存储算力一起自建，而不是用云，毕竟和云成本相比，对，尤其是加上 egress 之类的，所以你懂的。

that's a good idea but it also comes comes to there there are some of its own challenges

这是个好主意，但它也会带来一些它自己的挑战。

of course of course

当然，当然。

yeah like people who build the GPU data centers they might not expect this much uh storage and yeah people build storage typically they just build it somewhere.

对，比如那些建 GPU 数据中心的人，他们可能没料到需要这么多呃存储，而且，对，建存储的人通常就是随便找个地方建。

There's just CPUs.

那里就只有 CPU。

I just looked up five uh AWS only charges for egress, not ingress.

我刚查了一下，呃 AWS 只对 egress 收费，不对 ingress 收费。

Tier five for five pabytes is 230K.

5 PB 对应的第五档是 23 万美元。

Yeah, even more expensive.

对，还要更贵。

But storing is per month, right?

但存储是按月收的，对吧？

You check in and you cannot check out.

你存进去，然后取不出来。

Uh so it's cool.

呃所以还行。

It's okay.

还好。

So

所以——

TLD, you know, my my math is larger than you think.

总之，你知道，我这笔账比你想的要大。

Yes.

是的。

Yeah.

对。

My backhand math of GPU hours times GPU cost is also very much, you know, I'm missing some storage.

我那笔粗算——GPU 小时乘以 GPU 成本——也很可观，你知道，我漏算了一些存储的部分。

So you're basically like also more IO bound than normal training.

所以你基本上比常规训练更受 IO 制约。

Cuz like data loading caching everything becomes super important.

因为像数据加载、缓存这些都变得极其重要。

Yeah.

对。

So in Cosmos we did a lot of optimizations to make it not IO bound.

所以在 Cosmos 里我们做了很多优化，让它不受 IO 制约。

So um yeah speaking of the training actually training the model the GPU cost.

所以嗯，说到训练，其实训练这个模型的 GPU 成本。

If you look up like the the open source model how big these video models are.

如果你去查一下那些开源模型，看看这些 video model 有多大。

think like LTX has 19b parameters that's a dense model and people are also exploring uh so it might be like a 20B active and uh like 100b uh total so that's that's even that's similar size as mediumsiz LM models and if you if you look at number of tokens we disclose that in Cosmos it's also like tens of trillions of tokens on the facial tokens.

我记得像 LTX 有 190 亿参数，那是个 dense 模型，大家也在探索，呃，所以可能会是 200 亿 active、呃 1000 亿 total 这样，所以这甚至……这跟中等规模的 LLM 模型体量相当，而且如果你看 token 数量，我们在 Cosmos 里披露过，也是几十万亿 token 这个量级的视觉 token。

So, putting this together, the cost of like training these video models, it's actually comparable with LMS.

所以把这些放在一起，训练这些 video model 的成本，其实和 LLM 是相当的。

Not to mention the the infra is slightly different from LM.

更别提基础设施跟 LLM 还略有不同。

So, it might be less efficient to train these models.

所以训练这些模型可能效率还更低一些。

Do you get the benefits of traditional diffusion speed up?

那你能享受到传统 diffusion 加速的那些好处吗？

So, for you know, images there's LCM, Lauras, for um you know, fine-tuning.

比如说，你知道，对图像有 LCM、Loras，用于嗯，你知道，微调。

There's there's a lot of stuff.

有很多这类东西。

Yeah, there's full matching.

对，还有 flow matching。

there's a lot of stuff that's been done.

已经有很多相关的工作做出来了。

Uh there's some overlap that applies to diffusion on the inference side and stuff or

呃有些在推理侧应用于 diffusion 的东西是有重叠的，之类的，或者……

Yeah.

对。

So, so the difference the inference side is a complete different story.

所以，所以区别在于，推理侧完全是另一回事。

I think for the training side it's might be a little bit hard to reduce that cost and for for the inference side the biggest gain is from the dissolation of these models.

我觉得在训练侧要压低那个成本可能有点难，而在推理侧，最大的收益来自这些模型的蒸馏。

So you you can it's called step dissolation slightly different from knowledge dissolution in all.

所以你可以，这叫 step distillation，跟通常说的 knowledge distillation 略有不同。

So you typically for flow matching models you need like 100 steps or something like uh diffusion model even need need even more like a thousand steps to to generate a good image or video.

所以通常对 flow matching 模型你需要大概 100 步左右，像 diffusion 模型甚至需要更多，比如一千步，才能生成一张好的图像或视频。

A step distillation is try to learn to generate few step from the model itself.

step distillation 就是试图学会从模型自身出发用很少的步数来生成。

It's kind of like now we you use the full model to generate in 100 steps and then you you take a model that only generate 10 steps and let that model learn from the perfect one.

有点像，现在我们用完整模型用 100 步来生成，然后你拿一个只生成 10 步的模型，让那个模型去向那个完美的结果学习。

Yeah.

对。

Uh why is this this work?

呃这为什么行得通？

Strong to weak seem kind of like strong to weak.

由强到弱，感觉有点像由强到弱。

I guess the from the modeling perspective the strong model the teacher model is trying to model the image and videos of inter internet and that distribution is extremely complex.

我想从建模的角度看，那个强模型、teacher 模型，是在试图对互联网上的图像和视频建模，而那个分布极其复杂。

But the step distill model is just trying to learn from the teacher.

但 step distill 后的模型只是在试图向 teacher 学习。

The teacher is is a model and the size is fixed.

teacher 是一个模型，尺寸是固定的。

the distribution is much simpler than the whole internet.

这个分布比整个互联网要简单得多。

That's the intuition I have.

这就是我对为什么 step distillation 能奏效的直觉。

Why step distillation can work?

为什么 step distillation 能奏效？

So usually these models serve in productions they only run in a few steps.

所以通常这些模型在生产环境里跑的时候只跑几步。

In in cosmos I believe we have we have like four step and eight steps.

在 Cosmos 里我记得我们有四步和八步的。

If you do some simpler task like image to image translation it can even run in first step like one step in in cosmos transfer.

如果你做一些更简单的任务，比如 image to image translation，它甚至能第一步就跑完，比如在 cosmos transfer 里一步就行。

Yeah, I think this is the same intuition that guides a lot of the consistency model work.

对，我觉得这跟指导了很多 consistency model 工作的直觉是一样的。

I sent you a a link for uh SCM.

我给你发了一个呃 SCM 的链接。

I don't know if you covered that.

我不知道你有没有看过。

To me, that was actually one of like the most impressive papers I've ever seen from OpenAI

对我来说，那其实是我见过 OpenAI 出的最令人印象深刻的论文之一。

that like uh this is the unifying grand concept of consistency models.

就像呃，这是把 consistency model 统一起来的宏大概念。

I don't know if you have any comments on this.

我不知道你对这个有没有什么看法。

So, so there are there are a few different approaches like uh

所以，所以有几种不同的路线，比如呃……

Oh, yeah.

哦，对。

Here it is.

在这儿呢。

Yeah.

对。

two steps versus 20 100 steps whatever it's already done.

两步对比 20 步、100 步之类的，反正已经搞定了。

So there there are a few different approaches for example consistency model and there were also actually we we shouldn't forget GAN so GAN actually that that was that was the OG uh step dissolvation cuz it it trend just one step to begin with.

所以有几种不同的路线，比如 consistency model，另外其实我们不该忘了 GAN，所以 GAN 其实才是那个最早的呃 step distillation，因为它一开始就是只用一步来训练的。

So actually a lot of uh for example there's a distribution matching distillation which use which uses GAN um as like as one of the laws for this solution.

所以其实有很多呃，比如有一种叫 distribution matching distillation，它用 GAN 嗯作为这个蒸馏的其中一个损失。

It GAN just tells you hey like generate an image and then it has a discriminator to to tell is this image real or not.

GAN 只是告诉你，嘿，生成一张图像，然后它有一个 discriminator 来判断这张图像是不是真实的。

So the model the model just need to learn one of the distribution not not the full distribution because in training the model is asked to reconstruct the ground truth image from the internet which is extremely hard and in when you're training GAN it's a one-step process.

所以这个模型只需要学习分布中的一个点，而不是整个分布，因为在训练时模型被要求从互联网上重建出 ground truth 图像，那极其困难，而当你训练 GAN 时，它是一个一步的过程。

It's just a hey you generate image does this image look look as real as the the image from the internet which is a much simpler task and yeah combining a lot of these approaches together people typically do that like consistency model and distribution matching and and again and we can get these few step models

它只是，嘿，你生成一张图像，这张图像看起来有没有像互联网上的图像那样真实，这是个简单得多的任务，而且对，把很多这些路线组合到一起，大家通常都会这么做，比如 consistency model 加 distribution matching 加 GAN，我们就能得到这些少步数的模型。

okay then there's one step I wanted to add which is audio

好，那还有一步我想补上，就是音频。

yes

是的。

and video Yeah.

还有视频，对。

So, uh, Gawk imagine 0.9 I I believe it's is the first first audio video trans model deployed at a large scale.

所以，呃，Grok Imagine 0.9 我我相信它是第一个大规模部署的音视频联合生成模型。

So,

所以，

and that was your first model.

那就是你们的第一个模型。

Yeah, that was the Grok imagine's first model.

对，那是 Grok Imagine 的第一个模型。

It's it's audio video uh joint generation.

它是音视频呃联合生成。

I think the the hard part is like the the modality alignment because before this joint model like we have we have text to video alignment we we have this uh corresponding text and video typically mo most of the VLMs they they understand images and videos very rare and they they don't understand audio mostly and if you look at the audio generation on the LM said you can talk to them perfectly fine but if you ask them to sing a song or something they typically is not very good also they don't have they don't have music either

我觉得最难的部分是模态对齐，因为在这个联合模型之前，我们有文本到视频的对齐，我们有对应的文本和视频，但通常大多数 VLM 它们理解图像和视频的情况很少，而且它们大多不理解音频；如果你看语言模型这边的音频生成，你可以和它们对话得很流畅，但如果你让它们唱首歌之类的，通常效果不太好，而且它们也没有，它们也没有音乐。

the hard part is that uh actually audio has two component it has like a a discrete component a continuous component the discrete component is like the the language so when we speak

难点在于，呃，其实音频有两个成分，它有一个离散成分和一个连续成分，离散成分就像是语言，也就是我们说话的时候。

it's just Um

它就只是，嗯

it's a ASR issue.

这是个 ASR 的问题。

Yeah.

对。

Yeah.

对。

It's it's text token with some characteristics I would say.

它就是带有一些特性的文本 token，我会这么说。

But uh but music

但呃但是音乐

I think the speech guys would disagree.

我觉得做语音的那帮人会不同意。

It's like discuencies and then you know I say largely the the but the music is completely different.

它有点像离散性的东西，然后你懂的，我说很大程度上，但音乐是完全不同的。

It's it's very continuous and you cannot model them like discrete tokens in language models.

它非常连续，你没法像语言模型里的离散 token 那样去建模它。

Uh this is like the the hard part for for model is uh not to mention we have to align text video and audio together.

呃这就是模型的难点所在，更别提我们还得把文本、视频和音频一起对齐。

So

所以

how

怎么

so significant some significant challenges are like so so first like we we talk about as the VLMs they cannot understand most of them cannot understand audio.

所以一些重大的挑战，比如说，首先就像我们说的 VLM，它们大多数没法理解音频。

So you have to have some way to to do the synthetic data generation for for audio.

所以你得有某种办法来为音频做合成数据生成。

You have to caption the model and that involve that involve sens data and human data effort a lot.

你得给模型做字幕标注，而这涉及大量的传感器数据和人工投入。

And not too surprisingly most of the OM are very bad at recognizing um like the the beat tone and the details of the of music.

而且不太意外的是，大多数模型在识别，嗯，比如节拍、音调和音乐的细节方面都很糟糕。

they they can they can give some general general prediction of which song is this but it's very hard to describe the details of of the music.

它们能，它们能给出这是哪首歌的一些大致预测，但很难描述音乐的细节。

Um like we mentioned in image generation like you have to describe image as details as possible so that someone blind can reconstruct it.

嗯，就像我们在图像生成里提到的，你得尽可能详细地描述图像，让一个盲人也能重建出来。

So here is like someone deaf can reconstruct how the music sounds like without without actually listening to it.

所以这里就是，让一个聋人能在不真正听到的情况下重建出音乐听起来是什么样的。

Maybe like you you can think of it need to have have the what do they call the the

也许你可以把它想成，需要有那个他们叫什么来着

you got to have have all the details of of the music and the dialogue.

你得掌握音乐和对白的所有细节。

So is the challenge there typically stuff like music and audio or is it just like is there a baseline okay there's enough data where we can understand you know narration conversation but there's nuances in audio that that's where you hit all the data issues or is it just from state zero you just do it all right.

那么这里的挑战通常是像音乐和音频这类东西，还是说，是不是有个基线——好，有足够的数据让我们能理解旁白、对话，但音频里有些微妙之处，正是在那里你撞上了所有的数据问题；还是说从零状态开始你就得把这一切全做对。

So one important thing is like the alignment.

所以有一件重要的事就是对齐。

So the model the model has to know like the video and audio the it has to have a time based alignment like at which time step the video and the audio token correspond to each other but we actually don't have these kind of alignment for for for most of the other modalities.

模型必须知道视频和音频，它必须有一个基于时间的对齐，比如在哪个时间步上视频和音频的 token 相互对应，但我们实际上对大多数其他模态并没有这种对齐。

If you think about like text text and image text and video they are loosely aligned.

如果你想想文本和图像、文本和视频，它们是松散对齐的。

So you can you can have a description of what's going on in the video, but you don't have to exactly uh you typically don't have exact description.

所以你可以有一段关于视频里正在发生什么的描述，但你不必精确到，你通常没有精确的描述。

Oh, at at the time step 1 second like what happened two seconds happened.

哦，在第 1 秒这个时间步发生了什么，第 2 秒发生了什么。

So what was the ideal time step?

那么理想的时间步是多少？

You have to ablate it and then it's like four seconds or something.

你得做消融实验，然后大概是四秒之类的。

So that comes down to how you design the model.

所以这归结于你怎么设计模型。

Yeah.

对。

to for the model to to be aware of as a time the time modality.

让模型能意识到时间这个时间模态。

So the model is like a time aware and that's something pretty unique if you think about all.

所以模型是时间感知的，如果你想想，这挺独特的。

So if you ask LM to complete a task, say they you ask them and they would say, "Oh, this tax task will probably take 12 hours to complete." And they they they come back in one hour.

所以如果你让语言模型完成一个任务，比如你问它们，它们会说，“哦，这个任务大概要花 12 小时完成。”然后它们一小时后就回来了。

They have already spent two days on this and have exhausted everything.

它们已经在这上面花了两天，把一切都耗尽了。

Yeah.

对。

So So the LMS themselves, they they don't have a sense of time there.

所以语言模型它们本身没有时间的概念。

I actually don't think that's just them not having a sense of time.

我其实不觉得那只是它们没有时间概念。

I think it's somewhat based, right?

我觉得它某种程度上是有依据的，对吧？

Like you tell someone, okay, go work on this feature, go implement this.

比如你告诉某人，好，去做这个功能，去实现这个。

There's a general understanding you would have of how long that would take without LLM's working LM speed, right?

对于这要花多长时间，你会有一个大致的理解——不是按语言模型的工作速度、语言模型的速度算，对吧？

So you think back like 2 years ago, if I tell you to like build me like a new front end for latent space, have a search bar, have all this, you'll estimate that it'll take a few days, right?

所以你回想两年前，如果我让你给 Latent Space 做一个新的前端，要有搜索栏，要有这一切，你会估计这得花几天，对吧？

So you tell an LLM, go build this,

所以你告诉一个语言模型，去做这个，

it'll take me a few days.

这得花我几天。

But, you know, I think it's somewhat grounded as opposed to them not having the best.

但你懂的，我觉得它某种程度上是有依据的，而不是说它们没有那种能力。

Not saying that they have a great understanding, but I think that example is like you can see where it comes from, right?

我不是说它们理解得很好，但我觉得那个例子能让你看出它是从哪来的，对吧？

You're trained on all over the

你是在整个……上面训练出来的

They're trying to estimate what a human would say.

它们是在试图估计一个人类会怎么说。

Yes.

对。

Because that's what the that's what the data kind of represents.

因为那就是数据所代表的东西。

Then the core on the internet people have a estimate.

然后核心是互联网上人们有一个估计。

Yeah.

对。

And not not even just in direct like training samples, right?

而且甚至不只是在直接的训练样本里，对吧？

just your world understanding of tokens of how long stuff takes, right?

就是你对 token 的世界理解，关于做各种事情要花多久，对吧？

Go read a book.

去读一本书。

It'll take you a while, right?

这得花你一阵子，对吧？

Even if you do nothing but read a book, it takes a few days.

哪怕你什么都不做只读一本书，也得花几天。

So, yeah, I'll read it.

所以，是的，我会读它。

It took me a few hours.

花了我几个小时。

It'll take me a few hours to go through this research.

读完这份研究得花我几个小时。

But some tangent,

不过有点扯远了，

somewhat a yeah, this is a train of thought I haven't really expressed until now is which is basically like a full world model must also be recursive, meaning that the participant in the world model must also be aware that they have a world model.

有点，对，这是一个我到现在都还没真正表达过的思路，基本上就是：一个完整的 world model 也必须是递归的，意思是 world model 里的参与者也必须意识到它们自己拥有一个 world model。

Uh, which is like this whole recursive thing down the down the line.

呃，这就像是一整套递归的东西一路往下延伸。

Um but yes and and that the world model can be wrong and that they need to update it and blah blah blah.

嗯，但没错，而且这个 world model 可能是错的，它们需要更新它，等等等等。

Yeah, we we've uh argued this on the uh newsletter as well that there needs to be sort of recursive or adversarial world models.

对，我们在 newsletter 上也论证过这一点，认为需要某种递归的或对抗式的 world model。

Okay, I mean just you know to ask how do you define world models?

好，我是想问一下，你怎么定义 world model？

Oh yeah, let's go there.

哦好，咱们就聊这个。

Yeah.

对。

So just for context, you know, we talked about uh video generation and then there's a if you say there's a distinction between world models.

先交代一下背景，我们前面聊了视频生成，然后你说 world model 和它之间是有区别的。

Um what's your what's your definition?

嗯，你的定义是什么？

How do you see the two?

你怎么看这两者？

Yeah.

好。

So so disclaimer, I'm not going to debate like what is world model.

先声明一下，我不打算去争论到底什么才叫 world model。

Yeah.

对。

Like there there are many definitions.

因为定义有很多种。

So I'll just talk about my definition since I I came from the multimodel multimodel domain.

所以我就讲我自己的定义吧，因为我是从 multimodel（多模态）领域过来的。

So mainly talking from video.

所以主要是从视频的角度来谈。

So world model is like real time interactive long horizon videos.

在我看来，world model 就是实时、可交互、长时程的视频。

So there there are three parts like so we let's talk about them one by one.

它有三个部分，我们一个一个来说。

So the so interaction so we just we just look at Facebook and neuro computer.

先说交互（interaction），我们就看看 Facebook 和 neuro computer（神经计算机）这类东西。

So the interaction part of it.

先说它的交互部分。

So you worth model can allow you to interact with them through keyboard mouse and maybe also voice.

你的 world model 应该允许你通过键盘、鼠标、也许还有语音去和它交互。

So the these all of all the modality you can you can interact with with the model and the model should respond reasonably.

所以这些所有的模态你都可以拿来跟模型交互，而模型应该做出合理的响应。

Second part is real time.

第二部分是实时（real time）。

So once you once say you you move your mouse like if say the the world model generate the game like how how fast can that game respond?

比如说你一移动鼠标，如果 world model 生成的是游戏，那这个游戏能多快做出响应？

So if you're like professional CSGO players might say oh you have to respond in sub

如果你是专业的 CSGO 玩家，可能会说，哦，你必须在 sub（低于）……

sub 10 milliseconds or or even less.

低于 10 毫秒甚至更短的时间内响应。

So that's not I guess most of the

所以那个不是……我想大多数……

No, 60 fps.

不，60 fps。

Let's go.

来吧。

Oh, 300 fps.

哦，300 fps。

500 FPS.

500 FPS。

Wait.

等等。

Uh, okay.

呃，好吧。

Yeah, I didn't do the math, but yeah.

对，我没算过，不过是的。

Okay.

好。

Uh,

呃，

yeah.

对。

300 FPS.

300 FPS。

That's a 3 millisecond.

那就是 3 毫秒。

So, you have to respond.

所以你必须做出响应。

Most of the video models cannot do that.

大多数视频模型做不到这一点。

Yeah.

对。

And but if you say if you have a video model that is say like a digital human the the response time might be more generous maybe like typically like for real time voice interaction is like 200 millisecond.

但如果说你有一个视频模型，比如说是个数字人（digital human），那响应时间可以宽松一些，一般像实时语音交互大概是 200 毫秒。

So that's uh uh that's much more generous.

所以那个呃就宽松多了。

But even 200 millisecond is pretty uh it is pretty tricky cuz like remember we mentioned you have this uh temporal compression coming from the VAE.

但即便是 200 毫秒也相当呃相当棘手，因为你记得我们提过，VAE 会带来时间维度上的压缩（temporal compression）。

So if you if you don't compress the temporal dimension your sequence lens is going to explode.

所以如果你不压缩时间维度，你的序列长度就会爆炸。

So if you want to have this real time real timeness in your model, you have to deal with long context problem.

所以如果你想让模型具备这种实时性，你就得处理长上下文（long context）问题。

And the third part is is long horizon because we are not going to just play with video games just like a few seconds.

第三部分是长时程（long horizon），因为我们玩视频游戏不会只玩几秒钟。

Most of video models only a few seconds.

大多数视频模型只有几秒钟。

We're going to play with minutes, hours.

我们要玩几分钟、几小时。

The model have to be able to generate long form content.

模型必须能够生成长篇内容。

So putting these three together, it's a real time long horizon interactive videos.

所以把这三点合在一起，就是实时、长时程、可交互的视频。

I think the the final state will be for example like a a video a video version of playbook where you can you can interact with a a neuro computer.

我觉得最终形态会是，比如说一个视频版的 playbook，你可以和一个 neuro computer 交互。

you you move your mouse and you you click on the generative interface and it will reply to you through through pixels generally in real time but getting there it's it's a very long way to get there so one of the first step uh at Gi Mine where I I led a small world model team there was was to build video extension so uh video extension uh it's the first step of interactivity.

你移动鼠标，点击那个生成式界面，它会实时地通过像素回应你，但要走到那一步还有很长的路要走。所以第一步之一，在 Gi Mine（xAI）我带一个小的 world model 团队时，就是去做视频延展（video extension），所以呃视频延展呃它是交互性的第一步。

It's a first step.

这是第一步。

Yeah.

对。

So it's the first step.

所以这是第一步。

We have it here.

我们这儿有。

Video editing.

视频编辑。

Yeah.

对。

[gasps]

[倒吸一口气]

Yeah.

对。

So the the first step is because uh this unlocks long horiz typically for most of the video generation models.

所以第一步是因为呃这解锁了长时程，一般来说对大多数视频生成模型而言。

You you give it a prompt or an image as a initial frame.

你给它一个 prompt 或者一张图片作为初始帧。

You generate video that's it.

你生成视频，就这样。

That's just uh one time done.

就是呃一次性搞定。

And some some creators would try to like use the last frame as a first frame for the second video.

有些创作者会尝试把上一段视频的最后一帧当作第二段视频的第一帧。

It kind sometimes it works but if you do it a few times it it says the call to a degree and it doesn't have that context over the full video.

有时候这管用，但如果你这么做几次，它就会在一定程度上（画质）崩坏，而且它没有覆盖整段视频的上下文。

So the temporal

所以那个时间上的……

Yeah.

对。

Cuz you only gave it the last frame.

因为你只给了它最后一帧。

Of course, right?

那当然了，对吧？

Yeah.

对。

Exactly.

正是如此。

But it's actually a pretty fun hack.

不过这其实是个挺好玩的 hack。

Like if you seen like

比如你有没有见过……

Oh, no.

哦，不。

He has something better.

他有更好的办法。

Yeah.

对。

Yeah.

对。

Yeah.

对。

And for example like view uh I remember view 3 has like uh a 1 second context of the the last video.

比如说 view（Veo）呃我记得 Veo 3 有大概呃 1 秒的上一段视频的上下文。

It it's slightly better than using the last frame but it has the same problem similar problem that it uh the quality would degree like if you extend a few times to like one minute the the video quality would look much worse than the first video.

它比只用最后一帧稍微好一点，但有同样的问题、类似的问题，就是呃画质会下降，如果你延展几次到大概一分钟，视频画质会比第一段视频看起来差很多。

Second another problem is that the model doesn't have long range knowledge of like what's happening before.

第二个另外一个问题是，模型没有关于之前发生了什么的长程知识。

So if they generate some dialogue some to people speaking and their voice might change over over some time especially if the one second conditioning it does not cover the previous context.

所以如果它们生成一些对话、一些人在说话，那些人的声音可能会随时间变化，尤其是如果那 1 秒的条件（conditioning）没有覆盖到之前的上下文。

So the these are the core challenges.

所以这些就是核心的挑战。

So the Gawk imagine video extension is has historical context of all of the previous generated videos.

所以 Grok Imagine 的视频延展是拥有之前所有已生成视频的历史上下文的。

It can uh it has it has a context of uh who's speaking and what what objects have appeared and everything having that to generate the next video.

它能呃它拥有呃谁在说话、出现过哪些物体等等的上下文，把这些拿来生成下一段视频。

So if we naively do this you you can imagine like just put all the previous history video tokens into the context the context lens will easily explode especially for for video models that that can be like a few a few million context I would imagine with that

所以如果我们很朴素地这么做，你可以想象，把之前所有历史视频的 token 全塞进上下文，上下文长度很容易就爆炸了，尤其是对视频模型来说，我想那可能会达到几百万的上下文。

yeah for example like in cosmos I think just 5 seconds of video is like a 50 50k or 60k number of tokens.

对，比如说在 Cosmos 里，我想仅仅 5 秒的视频就大概是 50k 或 60k 个 token。

So like if you do if you do 50 seconds as a 5 500k tokens, if you you do longer than that easily explode this long horizon uh problem with the first step we're trying to solve world model.

所以如果你做 50 秒就是 500k 个 token，如果你做得比这更长，这个长时程问题就很容易爆炸，我们用第一步来尝试解决 world model。

It turns out people Yeah, people love video extension like a lot a lot of creators love using video extension to create longer form videos.

结果发现大家……对，大家很喜欢视频延展，很多很多创作者都爱用视频延展来创作更长篇的视频。

This is the part I I like that you have a you have an intermediate step toward the the final goal instead just a straight straight shot to the the final version very much.

这是我喜欢的部分，就是你有一个通向最终目标的中间步骤，而不是一步到位直接冲向最终版本，非常喜欢这一点。

Yeah.

对。

But I can see you have a strong vision of where we want to end up.

但我看得出来你对我们最终要到达的地方有很强的愿景。

Does it seem like it's an efficiency issue like okay we're at a few million tokens context.

这看起来是不是一个效率问题，比如说好，我们现在处在几百万 token 的上下文。

You know, if you draw the parallel to language models, we had very short context, 2,000, 8,000, then you know, you scale it up 1 million, 10 million.

你知道，如果你拿语言模型来类比，我们一开始上下文很短，2000、8000，然后你把它扩到 100 万、1000 万。

Uh, sure, there's effective context.

呃，当然，是有有效上下文（effective context）这回事。

Um, you know, but at the end of the day, it's just what's it worth?

嗯，但归根结底，问题只是这值不值得？

Like, sure, there's a whole training data side.

比如说，当然，还有一整块训练数据的问题。

In video, it might be slightly easier cuz we have 100 million token video, right?

在视频里可能稍微容易点，因为我们有 1 亿 token 的视频，对吧？

Just take a movie with the full context there.

随便拿一部电影，完整的上下文就在那儿。

like is this efficiency from an inference standpoint that like it's expensive but we know how to solve it or like why is this not the approach?

那么这是不是从推理（inference）角度看的一个效率问题，就是它很贵，但我们知道怎么解决？还是说，为什么这不是那个（正确的）路子？

So like my broader point was on your second point of role models you say it needs to be interactive and live right you should be able to play a game and see the interaction live.

所以我更大的一个点是关于你说 world model 的第二点，你说它需要是可交互的、实时的，对吧，你应该能玩一个游戏并实时看到交互。

So one thing I see with research is a lot of what you actually serve is different than what you build.

所以我在研究里看到的一件事是，你实际部署（serve）的东西往往和你训练出来的东西不一样。

Right?

对吧？

So we talked about distillation.

所以我们聊过蒸馏（distillation）。

You train big model, you distill it, you do quantization, speculative decoding, we do all this stuff to serve it efficiently.

你训练一个大模型，把它蒸馏，做量化（quantization）、投机解码（speculative decoding），我们做这一大堆事情来高效地部署它。

Should we not just have a solution like a world model that can interact well do inference optimization, serve it, distill it secondary?

我们是不是就应该先搞出一个能好好交互的 world model 这样的方案，然后再把推理优化、部署、蒸馏放到第二位？

So make it real time after you solve it.

所以先解决它，之后再让它变实时。

So like another parallel is say continual learning right what we need is someone to solve it and show it works inefficiently give it a few years people will make it efficient same thing with regular attention right it worked over a few years people have different forms of attention and we've scaled it to be efficient at log context you know so kind of two things there right one is like it seems like it works you've scaled it can we not just scale it a lot more efficiently over time.

所以另一个类比是，比如说持续学习（continual learning），对吧，我们需要的是有人先解决它、证明它管用，哪怕效率低下，再给几年时间，大家就会让它变高效，普通的注意力（attention）也是一样，对吧，它先能用了，几年下来大家搞出了各种不同形式的注意力，我们把它扩展到能在长上下文（log context，即 long context）下高效运行，对吧，所以这里其实有两点，一是看起来它管用、你也扩展了它，那我们是不是就可以随时间把它扩展得高效得多。

Do we need a separate approach if this works?

如果这个路子管用，我们还需要一个单独的方案吗？

And same thing with interaction, right?

交互也是同样的道理，对吧？

Um if we can get it done like if we can solve some way that it works, you know, we can solve making it more efficient from an inference standpoint later.

嗯，如果我们能把它搞定，比如说如果我们能用某种方式解决让它管用，你知道，我们之后可以再从推理角度去解决让它更高效的问题。

Yeah, that's actually a very good point.

对,这其实是个很好的点。

So in videos there's actually a lot of redundancies.

视频里其实存在大量冗余。

So we we solve a lot of the pixel redundancy from VA, but there's more redundancy in long long range and long horizon videos.

我们用 VAE 解决了很多像素层面的冗余,但在长距离、长时程的视频里还有更多冗余。

Say if if a character appear in the first clip and then it disappeared.

比如说,如果一个角色在第一个片段里出现,然后消失了。

It only reappear like at the end of the video, you probably don't need the the context like in the middle of the generation.

它可能到视频结尾才重新出现,那你在生成中间部分的时候,可能根本不需要那段上下文。

So you you only need that character where where you need.

所以你只在需要那个角色的地方才用到它。

So that's why uh I help build another feature.

所以这就是为什么呃我帮忙做了另一个功能。

It's a reference to video.

它是一个 reference to video(参考图生视频)。

Is it here?

在这儿吗?

Is it the same same model release or different one?

它是同一次模型发布,还是另外一个?

It's a different one.

是另外一个。

You probably need to search on F reference to video.

你可能得搜一下 reference to video。

Okay.

好的。

So reference video allow you to like upload up to seven images as condition and generate a video.

reference video 允许你上传最多七张图片作为条件,然后生成一段视频。

Say like I want it can it can be characters or objects or even scenes.

比如说我想要——它可以是角色、物体,甚至是场景。

So like I I want I want condition on Sean's selfie and holding holding a blade.

比如说我想以 Sean 的自拍为条件,让他手里拿着一把刀。

You put the dog in the thing.

你把那只狗放进画面里。

Yeah, you can put them there and the video models will generate the the video from and copy the context over.

对,你可以把它们放进去,视频模型会据此生成视频,并把这些上下文复制过去。

So that can solve a lot of the problems there like the the long context problem.

所以这能解决很多那类问题,比如长上下文问题。

It doesn't need to have a very long context, but it's I feel like it's a intermediate solution.

它不需要有一个非常长的上下文,但我觉得这算是一个中间方案。

Yes.

是的。

The model should be able to like selectively know like where where should I draw

模型应该能够有选择地知道,该在哪里去取

references.

参考。

So say if I want to generate movie I generate it autogressive like a 10 second at a time or something.

比如说,如果我想生成一部电影,我是自回归式地生成的,一次生成 10 秒左右。

And now this character appear I can look back to where it first appear and uh bring that back.

现在这个角色出现了,我可以回看它第一次出现的地方,呃把那部分带回来。

Yeah this one I I put the references.

对,这一个我就放了参考图。

Yeah, that's uh Optimus Einstein myself Annie.

对,那是 Optimus、Einstein、我自己、Annie。

Oddly enough, I I used Grok search to find it and it pulled your LinkedIn post, but you know, we found it.

说来奇怪,我是用 Grok search 找到它的,结果它翻出了你的 LinkedIn 帖子,不过嘛,反正我们找到了。

But okay, this is a problem.

不过好吧,这是个问题。

This is not your fault, but like XI doesn't communicate all this work that you do very well because they just have the model release and then that's it.

这不是你的错,但比如说 xAI 没有把你做的这些工作很好地传达出去,因为他们就只是发个模型,然后就没了。

But like actually these details are very very good.

但其实这些细节非常非常好。

As far as I understand, everything you just described is state-of-the-art.

据我理解,你刚才描述的一切都是最前沿(state-of-the-art)的。

Like no one else has done it.

比如说没有别人做到过。

Thanks.

谢谢。

A lot of Yeah.

很多——对。

I I

我,我

And then And then you just put this blog post with the cookies.

然后你就发了这么一篇配了 cookies(小饼干示意图)的博客。

I'm like this is not enough, you know.

我心想,这可不够啊,你懂吧。

Uh but obviously this is like the high level numbers that people want to know.

呃但显然,这只是大家想知道的那些高层次数字。

But you know, like part of that is also like uh some some labs don't share research research into what happens.

但你知道,这其中一部分也是因为呃有些实验室不会把内部发生的研究细节分享出来。

No, but this is literally bragging about how good they are,

不对,但这明明就是在炫耀他们有多厉害啊,

right?

对吧?

Like why would you not say that you are capable of extending with full context, you know?

比如说,你为什么不说你有能力做到全上下文扩展呢,你懂吗?

This is not a secret sauce.

这又不是什么独门秘方。

This is like we did the work like Yeah.

这就是——我们把活儿干了嘛,就像——对。

I don't know.

我也不知道。

Yeah.

对。

I guess uh different labs have slightly different communication styles.

我想呃不同的实验室有稍微不同的传播风格吧。

Yeah.

对。

Anyway, if anyone from xAI is listening, we we are always happy to help you tell your story.

总之,如果 xAI 有人在听的话,我们随时乐意帮你们讲好你们的故事。

Yeah.

对。

Okay.

好的。

So, you did references and I think I think kind of the the point you're making is like this sort of like a cluge, right?

所以,你做了 references(参考功能),我觉得你想表达的点大概是,这某种程度上是个权宜之计(kludge),对吧?

like this is you can do seven but what about 100?

就是说你能做到七张,但一百张怎么办?

Right.

对。

Then you need a completely different thing.

那你就需要一个完全不同的东西了。

So I think it's like this is like a mechanism to like select the context from the history and you might not put the entire history into the context.

所以我觉得,这就像是一种从历史里挑选上下文的机制,你可能不会把整段历史都放进上下文里。

uh for example there's a paper called frame pack which have a heristic that the latest history like the last one second I put the the entire history and the history before that I would uh compress it and make the video smaller.

呃比如说有一篇论文叫 frame pack,它有一个启发式规则:对于最近的历史,比如最后一秒,我放入完整的历史;而更早之前的历史,我会呃把它压缩,把视频做得更小。

So I follow this pattern this beautiful pattern that the maximum sequence length is fixed.

所以我沿用了这个模式,这个很漂亮的模式——最大序列长度是固定的。

So the further you are from the current frame you have a smaller image.

所以你离当前帧越远,你的图像就越小。

So this is just a heristic.

所以这只是一个启发式规则。

I think it can be more automatic.

我觉得它可以做得更自动化。

The model is aware like which which history part of it can be select.

模型自己就能意识到,历史中的哪一部分可以被选取。

So this part of the research is actually being actively uh worked on by a lot of people.

所以这部分研究其实正被很多人积极地呃投入研究。

It's also quite interesting.

这也挺有意思的。

I feel this is actually this part of long context is a little bit ahead of the all part.

我感觉这部分——长上下文这部分,其实比其他所有部分都要领先一点。

So for example like in in LMS if you the context keep growing let's say if you you call tool and the tool call history is extremely long that's still in context and and keep growing keep growing even if you switch the topic to something else the whole context would there there are some agentic harnesses that help you to say prune the tool results and prune like when you when you query a file only show like the top 200 lines or something but those were very heristic driven

所以比如说在 LLM 里,如果上下文不断增长,假设你调用工具,而工具调用的历史极其长,那它还是留在上下文里,不断增长、不断增长,即使你把话题切换到别的东西上,整个上下文也还在那儿——有一些 agentic harness(智能体框架)会帮你,比如说修剪掉工具的结果,以及比如说当你查询一个文件时只显示前 200 行左右,但那些都是非常启发式驱动的。

for listeners we did a write up on the cloud code uh leak where there are eight different kinds of pruning uh including like you prune the tool results and all that so you can you can read up on that kind of thing

给听众们说一下,我们写过一篇关于 Claude Code 泄露的分析文章,里面有八种不同的修剪方式,呃包括比如修剪工具结果之类的,所以你们可以去读一下那类内容。

yeah I think uh one breakthrough in contining might be like a way to automatically uh manage it own

对,我觉得呃 context(上下文)方面的一个突破,可能会是一种能自动呃管理它自身的方式。

these are all heristics and they will be replaced by machine learning Yes, interestingly the same thing is being researched in both LMS and video models.

这些全都是启发式规则,而它们会被机器学习所取代。是的,有意思的是,同一件事情正在 LLM 和视频模型两边同时被研究。

So the interesting thing is also like in the paper you showed it's actually happening at the model level right compared to like language model sure we have base attention but you know we'll do our own compression we'll do our own pruning which is separate from model error eventually it all just boils in hopefully.

所以有意思的地方还在于,就像你展示的那篇论文里,这其实是发生在模型层面上的,对吧?相比之下,就像语言模型——当然,我们有基础的 attention,但你懂的,我们会做自己的压缩、做自己的修剪,那是独立于模型本身之外的,不过最终这一切希望都能融为一体。

Yeah.

对。

Yeah.

对。

I I think this is a form of like attention uh but like also no sort of reasoning attention.

我,我觉得这是一种 attention 的形式,呃但也算是某种推理式的 attention。

I I feel like that's different than normal attention.

我,我感觉那跟普通的 attention 不一样。

Is that does that make sense?

这么说讲得通吗?

Yeah.

对。

Yeah.

对。

It's it's different in the sense that attention not to mention uh set sparse attention aside like normal attention

它的不同之处在于,attention——先不提呃 sparse attention(稀疏注意力),就说普通的 attention

like you have to attend to all of the tokens.

就是说你必须去关注所有的 token。

Yes.

对。

So you you don't have a high level mechanism to to drop which tokens do you don't want to attend to as humans humans attention span is surprisingly small.

所以你没有一个高层次的机制来决定丢弃哪些你不想关注的 token,而人类嘛——人类的注意力跨度小得惊人。

Yes,

对,

you can only remember 11 digit of a phone number.

你只能记住电话号码的 11 位数字。

But I have feature detection, right?

但我有特征检测,对吧?

I can detect.

我能检测出来。

Oh, that's a sequence of 1 2 3 4 in a phone number that is 11 digit.

哦,这是一个 1 2 3 4 的序列,在一个 11 位的电话号码里。

Yeah, very good pattern matchers.

对,非常擅长模式匹配。

But humans contacts can like attention can work because we can dynamically pull in contacts from from different places.

但人类的上下文之所以能像那样运作——注意力之所以能运作,是因为我们可以动态地从不同的地方把上下文拉进来。

The same mechanism I think is going to happen for RMs and video models.

我觉得同样的机制也将会发生在 RLM(推理语言模型)和视频模型上。

Yeah, RLMs is recent is some of the recent work.

对,RLM 是最近的——是最近的一些工作。

Yeah,

对,

is there which is not that uh crazy, but it's just recursive.

它——那个其实也没那么疯狂,只不过它是递归的。

I think it's somewhat inherent in models too, right?

我觉得这在模型里某种程度上也是内在固有的,对吧?

Like

比如说

here's a nice example.

这儿有个不错的例子。

You pull up these, you can read it fine, but uh language models are also very good at slop parsing.

你把这些调出来,你能正常读懂,但呃语言模型也非常擅长解析这种含糊杂乱的文本。

Uh you know, you have a

呃你懂的,你有一个

I throw my typos in there.

我把我的错别字扔进去。

It doesn't matter.

没关系。

Yeah.

对。

Yeah.

对。

You have a you have a transcript, you have whatever.

你有一份转录稿,你有随便什么东西。

Just throw it in and it's very good at parsing through noise.

直接扔进去,它非常擅长从噪声中解析出内容。

um you know that may be a brute force.

嗯你知道,这也许是一种暴力破解。

It can look over it, reason over it, but like you know there's there's parallels to both.

它可以把它扫一遍、在上面做推理,但你懂的,这两边是有相似之处的。

I think it's just really fascinating how you relate the world model stuff to the video generation which I don't think a lot of people hear directly uh from people like like you.

我觉得,你把 world model 那套东西和视频生成联系起来的方式真的很迷人,而这种直接从像你这样的人口中听到的东西,我觉得很多人是听不到的。

So I think that's really helpful.

所以我觉得这真的很有帮助。

Any other work do we cover like video audio uh world models any other stuff in that omni team I guess or any other work at xAI you want to talk about?

还有别的工作吗——我们聊了视频、音频呃 world model,那个 omni 团队里还有别的什么吗,或者 xAI 还有别的什么工作你想聊的?

Seems like everything we see publicly announced.

感觉我们看到的都是公开宣布的那些。

Oh, cool.

哦,酷。

Cookies and then there's so much more to any underrated stuff, you know, just at the time there.

cookies(小饼干图),然后还有更多的——有什么被低估的东西吗,你知道的,就当下在那边的。

Yeah, I feel the is the culture is quite interesting and a bit underrated.

对,我觉得那个文化其实挺有意思的,而且有点被低估了。

So, so the culture is the culture is it's three sentences.

所以说,那个文化其实就是三句话。

move fast, build no goal is too ambitious.

快速行动,没有目标是过于野心勃勃的。

And the first principle like usually the the goal set was very ambitious.

还有 first principle,通常来说定的目标都非常有野心。

It wasn't very it wasn't it wasn't possible to to achieve when when I when I was thinking first thinking about it like for example I could build build something in in 3 months and was that like okay we're starting team we want image we want video do it by this deadline or you know how do you work back like was it just okay we have a rough by you know this date we want something out or is this like

在我最开始思考的时候,那些目标是不太可能实现的,比如说我能在 3 个月里做出点东西,当时是那种,好,我们组建团队,我们要 image、我们要 video,在这个 deadline 前做出来,或者说你们是怎么倒推的,是那种,好,我们粗略地在某个日期前想要有点东西出来,还是说这种

that's a very good point so it's from first principle thinking.

这是个非常好的点,所以这是从 first principle 思考出发的。

If you think about people might say like first principle thinking applied more to the physical world than the the models.

如果你想一下,有人可能会说 first principle 思考更适用于物理世界,而不是这些模型。

Uh I would say for example like if you think about some limitation for example acquiring data like how how fast can we acquire the videos and if if you think about training the models like uh what's the iteration speed for training a model end to end and how how would adding more GPUs accelerate that timeline and maybe if you need human data like what's the turn around time for for human data to arrive.

呃,我会说,比如你想一下某些限制,比如获取数据,我们获取视频的速度能有多快,再比如你想一下训练模型,呃端到端训练一个模型的迭代速度是多少,增加更多 GPU 又会怎样加速这个时间线,还有假如你需要人类标注数据,那这些人类数据到位的周转时间是多久。

If you put all of those together that is first principle thinking where oh you know like what is the timeline or the minimum number of days that is possible to achieve something

如果你把所有这些放到一起,这就是 first principle 思考,就是,哦,你知道的,达成某件事情可能的时间线,或者说最少需要多少天

I think there this is a lot of Elon's type of thinking right he's like I think he's famous for saying that the only law you can't break is the laws of physics something like that

我觉得这里面有很多 Elon 那种思维方式,对吧,他就像,我觉得他有句名言就是说,你唯一不能打破的定律就是物理定律,大概是这么说的

just broadly you you worked a lot with Elon

宽泛地说,你跟 Elon 有很多合作

yeah I I guess one benefit is like uh working at xAI, you got a chance to interact more with Elon.

对,我想有一个好处就是,呃在 xAI 工作,你有机会更多地跟 Elon 互动。

So, so I I was very fortunate to get a few retreats from him and that that was quite fun.

所以我很幸运参加了他的几次 retreat,那挺有意思的。

And uh he he also worked very closely uh with with people like like people imagine online like he he's very hands-on.

还有呃他也非常紧密地跟人合作,就像大家在网上想象的那样,他非常亲力亲为。

There are two things.

有两件事。

Um, one, so I was actually looking up uh Elon retweeting you.

嗯,一个是,我其实在查 Elon 转发你的那条推。

I'll pull it up.

我把它调出来。

Uh, he talked about you you tweeting that you have a really good voice mode.

呃,他提到你发推说你有个非常好的 voice mode。

I don't No, no, no him.

我不是,不不不,是他。

Oh, I also did it.

哦,我也做过。

But anyway,

不过总之呢,

I actually So, I would DM you feedback on voice mode cuz I was like, "Wow, really good." And then I'm like, "Oh, this sucks." But, um, I don't know.

我其实会私信给你反馈 voice mode,因为我当时就想,"哇,真不错。"然后我又想,"哦,这也太烂了。"但是,嗯,我也说不好。

Anything you want to talk about about your voice mode building it?

关于你做 voice mode 有什么想聊的吗?

Was it a team you worked on as well?

那也是你参与的一个团队吗?

Was that that's actually not a part of the the team I worked

那其实不是我参与的那个团队

on?

吗?

Probably worked on more the video.

我大概更多是做 video 那块。

No.

不是。

Uh but Grock voice actually like very good.

呃不过 Grok voice 其实做得非常好。

I this one of those things where like uh first of all you can speak at 2x which is fun.

我觉得这是那种,首先呢,你可以用 2 倍速说话,这挺好玩的。

Uh which I listen to 2x so I like to speak at 2x but also I think like the interruption was better than Gemini.

呃我听东西都是 2 倍速,所以我喜欢用 2 倍速说话,不过我还觉得它的打断比 Gemini 做得好。

Uh I don't know how it compares to chatbt real time now.

呃我不知道它现在跟 ChatGPT 的实时模式比怎么样。

But like you know as far as like driving was concerned like having Grock in my Tesla and like driving I think it was like it's really good experience.

但是,你知道的,就开车这件事而言,在我的 Tesla 里用 Grok 然后开车,我觉得这是那种,真的是非常好的体验。

Yeah.

对。

He likes voice mode but also um just the crazy reach by

他喜欢 voice mode,不过嗯还有那个疯狂的触达量

50 million views are just saying yes truth

5000 万次观看,就说个 yes 而已,真的

true.

真的。

Um

嗯

oh my god.

我的天。

But uh you know it's it's pretty cool how fast it came out.

但是呃你知道的,它出来得这么快,挺酷的。

I guess the other thing is the safety aspect of video mode.

我想另一件事是 video mode 的安全性方面。

Anything interesting to talk about there?

这方面有什么有意思的可以聊的吗?

So

所以

spicy

辛辣

spicy question.

辛辣的问题。

A lot of the countries where they they don't allow like a generative data generative AI videos without watermarks.

很多国家是不允许生成式 AI 视频不带水印的。

So in all of the those countries uh Grok imagine had watermarks and a lot of the a lot of the takedowns of if of the videos were also happening extremely fast.

所以在所有那些国家,呃 Grok Imagine 都会带水印,而且很多很多视频的下架处理也发生得极快。

I mean it's it's part of running a social platform but also it transfers nicely to the geni side.

我的意思是,这是运营一个社交平台的一部分,不过它也很好地迁移到了 geni(生成式 AI)这一侧。

Do you have a perspective on synth ID versus other kinds of watermarking?

你对 SynthID 相较于其他类型的水印有什么看法吗?

Yeah, I guess it's going to be yeah it's going to be harder and harder to to detect is

对,我想这会,对这会越来越难检测出来

yeah these things.

对,这些东西。

So since ID one thing is uh previously it was only Google

所以说 SynthID,有一点是呃以前只有 Google 在用

and now like a lot of different labs were also adapting it.

而现在很多不同的实验室也在采用它了。

It's a limitation is like the the technology is the paper was out there and people can reverse engineer like how to get rid of it.

它有一个局限,就是这个技术的论文已经公开了,人们可以逆向工程出怎么去掉它。

Yes.

是的。

And it's I think even as it advance it's it's still still possible to reverse engineer it.

而且我觉得即便它不断进步,还是还是有可能被逆向工程破解的。

Yeah.

对。

Uh so if you are interested you can go on to Reddit and people have taken out the exact like uh I don't know what do you call it mask or

呃所以如果你有兴趣,你可以上 Reddit,有人已经把那个确切的,呃我不知道该叫它什么,mask 还是

pattern that Google applies and then you can apply it onto any Google generated photo and you can reverse out the synth ID.

Google 应用的那个 pattern 给提取出来了,然后你可以把它套用到任何 Google 生成的照片上,你就能逆向去掉 SynthID。

And it's it's also harder and harder to just judge by eyes.

而且用肉眼判断也越来越难了。

I remember like a couple years ago there are like a six fingers or or something.

我记得像几年前那样,会有那种六根手指之类的。

It's very obvious.

非常明显。

My my current is actually the audio.

我现在(判断的依据)其实是音频。

I feel like the audio is really lacking.

我感觉音频这块真的很欠缺。

Uh my way to tell if something is the eye generated outside of like okay I think I've seen enough I have a decent eye.

呃我判断某个东西是不是 AI 生成的方法,除了那种,好吧我觉得我看得够多了,我眼力还不错之外。

The audio matchup especially of Sora is not great.

音频的匹配,尤其是 Sora 的,做得不太好。

It's all similar style but there's

都是同一种风格,但是有

you know those are minor imperfections.

你知道的,那些都是些小瑕疵。

I think the the point is that like actually my closest reference to this is also Ian Goodfellow because I think he did like the adversarial gam thing where like it's like okay here's a picture of a zebra then you like change one pixel and it becomes a panda

我觉得重点是,其实我对这个最贴近的联想是 Ian Goodfellow,因为我觉得他做了那个 adversarial GAN 的东西,就是那种,好,这里有一张斑马的图片,然后你改动一个像素,它就变成了熊猫

right this like a classic computer vision issue

对吧,这就像是一个经典的计算机视觉问题

yeah if you think about how how these models were were trained like I like I mentioned before like GAN was in the training process the objective again is you the model generate an image and the model there's a judge to tell like if the image is real or not.

对,如果你想一下这些模型是怎么训练的,就像我之前提到的,GAN 在训练过程中,它的目标又是,你让模型生成一张图片,然后有一个判别器来判断这张图片是不是真的。

The model is trained to make the image more real.

模型被训练得让图片更逼真。

So as the model become more and more advanced it's going to be harder and harder.

所以随着模型变得越来越先进,这会越来越难。

For me personally now I have to judge by if the these videos have logical sense the this video

对我个人来说,现在我得靠这些视频有没有逻辑上的合理性来判断,这个视频

have a role model.

有没有一个 world model(世界模型)。

Yeah.

对。

Yeah.

对。

Yeah.

对。

No, I I also like it the the the audio is too nice, like too st too studio quality, the lighting is too good, the skin is too clear, you know, basically the lack of imperfections.

不,我也是,我觉得那个音频太好听了,像太录音棚级别了,光线太好了,皮肤太干净了,你知道的,基本上就是缺少那些瑕疵。

Do we have a good way to do reasoning in diffusion?

我们有没有一个好办法在 diffusion 里做推理?

Like is that what separates video generators from world models or in you know we really know how to apply it to auto regressive language models?

比如,这是不是就是把 video generator 和 world model 区分开来的东西?或者说,我们其实真正搞明白的是怎么把它用在 auto-regressive 的语言模型上?

Is there a parallel for diffusion video gen world models like

对于 diffusion 视频生成的 world model,有没有一个类似的对应物,比如说

on that point right is he he has a thing on video agents.

说到这一点,他有一篇讲 video agent 的东西。

Yeah that's a good question.

对,这是个好问题。

Yeah, actually I have a I have a pretty big claim.

对,其实我有一个相当大的主张。

the uh the int the visual model technology is more mature the like mostly the gain comes from language model not not coming from the the v the video model itself like the the real distriution model themselves in cosmos like could be typically the these models they have two parts like there's a there's a prompt rewriter or the prompt up sampler part I think in uh in cosmos we use llama or we use mix throw and the cosmos video model itself is only 7B and the model the language model is a prompt rewriter is it's bigger than that.

呃,视觉模型这项技术更成熟了,大部分的提升其实来自语言模型,而不是来自视频模型本身——不是来自 Cosmos 里那些真正的 diffusion 模型本身。通常这些模型有两个部分:有一个 prompt rewriter,或者说 prompt up-sampler 部分,我记得在 Cosmos 里我们用的是 llama,或者用 Mixtral,而 Cosmos 视频模型本身只有 7B,而那个语言模型、那个 prompt rewriter,是比它更大的。

So the prompt writer's task is to take take a user instruction and convert it to extremely detailed description of the video.

所以 prompt rewriter 的任务,就是把用户的指令,转换成对视频极其详细的描述。

So because the video the visual the video diffusion models I would describe they're kind of dumb because they they take the input instruction literally because in the training process remember that we have to describe the video as as detailed as possible when we are creating the the synthetic uh text pair.

所以,因为视频、视觉、视频 diffusion 模型,我会形容它们有点笨,因为它们会照字面理解输入的指令——因为在训练过程中,记住,我们在制作合成的文本配对时,必须把视频描述得尽可能详细。

So this model they take those kind of instruction to generate the videos.

所以这些模型就是拿那样的指令去生成视频的。

So in when you're taking the user instruction the user instruction usually very are simple just say a cat or something.

所以当你拿到用户的指令时,用户的指令通常非常简单,就说一句"一只猫"之类的。

If you put a cat in in the video model they would take that instruction literally they would literally show a t a cat in maybe a white background because you didn't describe the background.

如果你把"一只猫"喂给视频模型,它们会照字面去理解那个指令,它们会真的就展示一只猫,可能是在白色背景里,因为你没描述背景。

the cat is not moving because you didn't describe it.

猫不会动,因为你没描述它动。

It it takes the instruction quite literally.

它相当照字面地理解指令。

It's kind of it's kind of dumb and the the prompt writer is actually a much bigger model which a language model that takes the the user instruction and expanded.

它有点笨,而 prompt rewriter 其实是一个大得多的模型,是一个语言模型,它接收用户的指令并加以扩展。

So the thinking process you mentioned uh is from there.

所以你提到的那个思考过程,就是从这儿来的。

So if if you if you look at like GPD image like you generate a image in 3 minutes 3 minute it's it's not all like a pixel generation a lot of time is spending in thinking

所以如果你去看比如 GPT image 那样的东西,你生成一张图要 3 分钟,这 3 分钟并不全是像素生成,很多时间是花在思考上的。

so so prompt rewriting now have evolved to like not only just thinking it it can it can also be a agentic model for example say you want you wanted to generate the image of today's news.

所以 prompt rewriting 现在已经演进到不只是思考了,它还可以是一个 agentic 的模型,比如说你想生成一张关于今天新闻的图片。

So the so it's likely we'll go to fetch today's news online and then like process them, dagger them, then organize the layout and generate it.

所以它很可能会去网上抓取今天的新闻,然后处理它们、把它们整理一下,再组织版面并生成出来。

Another thing quite interesting is um

另一件挺有意思的事情是,嗯

if I'm not mistaken these are it's no longer a diffusion model though, right?

如果我没记错的话,这些其实已经不再是 diffusion 模型了,对吧?

auto reggressively or is there still

是 auto-regressive 地生成,还是说仍然有

there are different approaches for example like uh chai omni since they said is omni I believe it's a it's a single model

有不同的做法,比如说呃 Chai Omni,既然他们说了是 omni,我相信它是一个单一的模型

maybe it's something like uh it's a language model with a diffusion head or something like the language model do the thinking do the agent take two calling and then it would uh use the diffusion head to generate the image in the end there there were also approaches like cosmos where you have a separate language model and separate distress models and there there are also like a purely language model like you you discretize the images and then you generate an image as discrete tokens.

也许它是类似于呃一个带 diffusion head 的语言模型之类的,由语言模型来做思考、做 agent 的工具调用,然后最后用 diffusion head 来生成图像;也有像 Cosmos 那样的做法,你有一个独立的语言模型和独立的 diffusion 模型;还有像纯语言模型那样的做法,你把图像离散化,然后把图像当作离散的 token 生成出来。

So there are different approaches.

所以有不同的做法。

I would say like

我会说,像

one of one of the claims I've seen for why these approaches struggle is because a lot of the benefits for how we currently learn reasoning with language models is you basically iteratively generate reason you have your thought and then you work on that answer.

我见过的一个关于为什么这些做法会遇到困难的说法是,我们目前用语言模型学习推理的很多好处,基本上在于你是迭代地生成推理——你有你的想法,然后你在那个答案上继续琢磨。

Right?

对吧?

So if you have like omnimodel and then diffusion head you can't feed that back in to continue reasoning.

所以如果你有一个 omni model 加一个 diffusion head,你没法把那个结果再喂回去继续推理。

Right?

对吧?

So you can't go like text image text image.

所以你没法做到 文本、图像、文本、图像 这样。

You can't reason on the output and then go back to diffusion.

你没法在输出上做推理,然后再回到 diffusion。

But I guess in the new Gemini Omni, you would be able to as long as you have diffusion.

但我猜在新的 Gemini Omni 里,只要你有 diffusion,你就能做到了。

Yeah.

对。

I'm not sure if they have that process.

我不确定他们有没有那个流程。

I guess it's definitely possible in the Omni paradigm.

我猜在 omni 这个范式下这肯定是可能的。

So if you think about like traditional multimodel language model, they would have a VIT encoder that can encode the image.

所以如果你想想传统的多模态语言模型,它们会有一个 ViT encoder 来编码图像。

So if they have a diffusion hat, they can generate the image and then put that back into the VIT encoder, encode that and then do do the iterative refinement if the result.

所以如果它们有一个 diffusion head,它们就能生成图像,然后把它放回 ViT encoder,编码它,然后对结果做迭代式的精修。

Yeah,

对,

I think you have to jointly train the VIT and the diffusion to make that somewhat reasonable because otherwise you're kind of like uh mismatching or feeding in slop.

我觉得你得把 ViT 和 diffusion 联合训练,才能让那件事变得多少合理一点,否则你多少会呃出现不匹配,或者喂进去一堆垃圾。

I think it depends on your stage of training you might be able to freeze it.

我觉得这取决于你训练的阶段,你也许可以把它冻结住。

But anyway also just on your earlier

不过总之,还是回到你前面说的

uh I I wanted to also make explicit we do know that nano banana and GPT image are auto reggressive uh language model with diffusion head uh as far as I can tell from your description of grog image it is not it is it is end to end

呃我还想把这点讲明白:我们确实知道 nano banana 和 GPT image 是带 diffusion head 的 auto-regressive 语言模型,呃而据我从你对 Grok Imagine 的描述来看,它不是,它是端到端的。

I I cannot yeah the way that you described it uh but I think there's there's different approaches right like you started off saying prompt rewriter is like a big part of the intelligence

我不能……对,照你描述的方式呃,不过我觉得有不同的做法,对吧,就像你一开始说的,prompt rewriter 是这份智能里很大的一部分

and even on that I think everyone should try using an early diffusion model.

而且就这一点,我觉得每个人都应该试试用一下早期的 diffusion 模型。

If you've used stable diffusion one or whatever, if you've seen the prompts like uh you know ultra high-res 4K this style like oh my god the first time I tried one you don't talk to them like language models right your prompting is very uh you know comma separated

如果你用过 Stable Diffusion 1 之类的,如果你见过那种 prompt,比如呃你懂的,ultra high-res 4K、这种风格,天哪,我第一次试的时候——你不是像跟语言模型那样跟它们说话的,对吧,你的 prompt 是很呃你懂的,逗号分隔的。

literally talking in the labels that were in the data set right

字面上就是在用数据集里出现过的那些标签来说话,对吧

but basically like I'm just trying to make the point that prompt writer and then image is different from auto reggressive language model with diffusion head right they're different things

但基本上,我只是想说明这一点:prompt rewriter 然后接图像,这跟带 diffusion head 的 auto-regressive 语言模型是不一样的,对吧,它们是不同的东西。

yes they're different

对,它们是不同的

just wanted to establish I say like the the common part is like the the image part.

只是想把这点确立下来,我想说,共同的部分是图像那部分。

So so it's it's quite surprising that like a lot of the improvement came from the

所以,所以挺让人意外的是,很多提升其实来自那个

language

语言

the thinking the the tool calling.

那个思考、那个工具调用。

So, so I still remember like in Cosmos I I generate a happy sheep and can if without any rewriting it's it looks so CGI and after rewrite it looks it it looks so beautiful I think

所以,我到现在还记得,在 Cosmos 里我生成一只快乐的羊,如果不做任何 rewriting,它看起来特别 CGI,而 rewrite 之后它看起来就特别漂亮,我觉得

without any joint training.

而且没有任何联合训练。

Yeah, actually without any joint training uh it's with rewriting it's already much better.

对,其实没有任何联合训练,呃光是加上 rewriting,就已经好很多了。

I think a very interesting thing I guess will happen is the video agents mostly language models they'll call these uh generative model either it's a separate model or diffusion head whatever as tool.

我觉得会发生的一件很有意思的事情是,video agent 大多是语言模型,它们会把这些呃生成模型——不管是一个独立的模型还是一个 diffusion head 之类的——当作工具来调用。

So this model can iteratively refine the results or or even like generate longer content through a very long train of thought.

所以这种模型可以迭代地精修结果,甚至可以通过一条很长的思维链生成更长的内容。

It's actually very similar to how human create art.

这其实跟人类创作艺术的方式非常相似。

So, so we don't we don't generate the pixels directly.

所以,我们不是直接生成像素的。

We we iterally draw something.

我们是迭代地画出一些东西。

Um, and I think through this process like the these models not only use diffusion diffusion as one of the tool.

嗯,而且我觉得通过这个过程,这些模型不仅把 diffusion 当作其中一个工具用。

It can also use traditional tool.

它还可以用传统的工具。

It can also use uh image editing tools from Photoshop.

它还可以用呃 Photoshop 里的图像编辑工具。

You can use uh video video editor, ffmpeg whatever to take combination of these and the generative AI technology as a as a set of tool and they can they can iteratively create a better much better uh video for for production grade quality.

你可以用呃视频、视频剪辑工具、ffmpeg 等等,把这些东西和生成式 AI 技术组合起来当作一套工具,它们就能迭代地创作出好得多的呃视频,达到量产级的质量。

If you look at existing u professional creators, they don't they don't end at uh generating a video from from these models.

如果你去看现有的呃专业创作者,他们不会在从这些模型生成一段视频后就停下来。

They will take this video to to their editor and edit here and there.

他们会把这段视频拿到他们的剪辑软件里,这儿改改、那儿改改。

So much post-prouction in sometimes actually like the the reason the video is good is not really the video model, it's actually the editing.

有太多后期制作的成分了,有时候视频之所以好看,其实并不是因为视频模型本身,而是因为剪辑。

And yes, we also are engaged in the same process as well.

对,我们也在经历同样的过程。

Would you love to use a video editing model?

你会想用一个视频剪辑模型吗?

Yeah, actually there's the Grok Grok imagine agent beta that that was the that was the first attempt in that direction.

对,其实有个 Grok imagine agent 的 beta 版,那就是朝这个方向的第一次尝试。

So I I think the like the process would be similar mode.

所以我觉得这个流程会是类似的模式。

Yeah.

对。

You can you can ask it to

你可以让它去

there's no blog post for it.

关于这个还没有博客文章。

maybe generate a a one minute uh video like which which is not possible if you ask the same prompt to video models but this model will iterally call different tools to do that.

比如生成一段一分钟的呃视频,如果你把同样的 prompt 丢给视频模型是做不到的,但这个模型会真的去调用不同的工具来完成它。

So yeah this is actually an interesting thing.

所以对,这其实是件挺有意思的事。

So when when we first release a video editing model like I see on X some people try the video editing feature with like edit this video to be one minute but because they didn't understand how video editing work.

我们第一次发布视频剪辑模型的时候,我在 X 上看到有些人尝试用视频剪辑功能,比如'把这段视频剪成一分钟',但因为他们不理解视频剪辑是怎么工作的。

Video editing typically is just a removal add replace transfer this kind of thing.

视频剪辑通常就是删除、添加、替换、转换这一类操作。

But that's actually a valid request under the assumption of video agents.

但在视频 agent 的设定下,那其实是个合理的请求。

So these agents should be able to understand this kind of uh long horizontally uh create a long form video.

所以这些 agent 应该能够理解这种呃长跨度的呃创作一段长视频的需求。

I think this is uh this is really fascinating because it it's kind of take it's taking the same uh same direction as first you have this uh uh assisted uh AI assisted coding kind of like tab completion and GitHub copilot and from there you gradually evolve to codeex and cloud code where you do things fully automated.

我觉得这个呃这个真的很迷人,因为它走的其实是跟当年一样的呃方向:一开始你有这种呃呃辅助的呃 AI 辅助编程,类似 tab 补全和 GitHub Copilot,然后从那儿逐步演进到 Codex 和 Claude Code,让事情完全自动化完成。

So in in Asian in gag imagin Asian mode you can you can still go in there and do do stuff by yourself but gradually as as the model capability increase it will be able to do everything fully automated.

所以在 agent 模式里,在 Grok imagine 的 agent 模式里,你仍然可以进去自己动手做点东西,但随着模型能力的提升,它会逐渐能够把一切都完全自动化地完成。

Yeah.

对。

Um I like that.

嗯,我喜欢这个。

Okay.

好的。

So it looks like it's still generating.

看起来它还在生成中。

Also I I did notice the crock midgen was always very very fast.

另外我确实注意到 Grok imagine 一直都非常非常快。

I don't know if this is something you guys benchmark but like this is just a tangent compared to uh when I used to use before the latest open image gen and same with Gemini nano banana I would often times use crop just for

我不知道这是不是你们会去 benchmark 的东西,不过这只是题外话——相比之下,呃在最新的开放 image gen 出来之前,还有 Gemini nano banana 也一样,我经常会用 Grok 就是为了

it's in the benchmark somewhere in imagine API blog post that they have all the the speed things uh it mostly combination of distillation plus inference

这个在 benchmark 里有,在 imagine API 的博客文章里某处,他们把所有速度相关的数据都放了呃主要是 distillation 加上 inference 的组合

yeah there there are a bunch of things like we talk about dillation If you talk about thinking if you don't have any thinking budget the model can just think 3 minutes and then come back to you and also like inference the inference infra team was very talented and they were they were able to accelerate a hell lot of these models.

对,有一堆因素,比如我们聊过的 distillation。如果说到 thinking,如果你不设任何 thinking budget,模型完全可以想上 3 分钟再回复你;还有 inference,inference infra 团队非常有才华,他们能够把这些模型加速一大截。

Yeah.

对。

Yeah.

对。

Yeah, I mean, you know, my comment on the on the video agents things, like I'm trying to figure out like when people say video agents, when you initially told me about your bet on video agents or your your vision for video agents, I was a little bit disappointed.

对,我是说,你知道,我对视频 agent 这件事的看法是,我一直想搞清楚,当人们说视频 agent 时——当你最初跟我讲你在视频 agent 上的押注、或者你对视频 agent 的愿景时,我其实有点失望。

I was like, h you mean like models are tapped out now, we have to do agents, but like I think you have to, right?

我当时想,呃你意思是模型现在已经到头了,我们只好去做 agent?但我觉得你不得不这么做,对吧?

The question now is how much model training is is it really going to make a difference versus just building a better harness?

现在的问题是,到底做多少模型训练才真的能带来差别,而不是仅仅去搭一个更好的 harness?

Like you said, uh the models don't have to be jointly trained.

就像你说的,呃这些模型不必联合训练。

Uh you can just take an offtheshelf Frontier reasoning model, slap it on a harness, give it Grock as a tool, that's it.

呃你完全可以拿一个现成的 Frontier reasoning model,把它套进一个 harness,把 Grok 作为工具给它,就这样。

That's your video agent.

那就是你的视频 agent 了。

Doesn't seem super satisfying.

听起来不太让人满足。

Obviously, you can co- train and and get some more percentage points of proper performance, but like if your central claim that the majority of video or generative media uh alpha or whatever is actually coming from language intelligence and not um image diffusion or video diffusion, then that is the future.

当然,你可以联合训练,再多榨出几个百分点的实际性能,但如果你的核心论点是:视频或者说生成式媒体的呃大部分 alpha 之类的东西其实来自语言智能,而不是嗯 image diffusion 或 video diffusion,那么那就是未来。

It's primarily just wait.

那基本上就只是等待而已。

If you pop back at the example, you know, it it generated frames.

如果你切回那个示例看看,你知道,它生成了一些帧。

Sorry to interrupt, but you know, it's been saying like, okay, I'm going to start stitching these frames together.

抱歉打断一下,不过你知道,它一直在说,好,我要开始把这些帧拼接起来了。

It's using ffmpeg using

它在用 ffmpeg,用

this is what GBC image pro as well is doing, right?

这也正是 GPT image pro 在做的事,对吧?

Like it's it's also just writing code in the background and then just

它也就是在后台写代码,然后就

stitching

拼接

doing an image pass on the final output.

对最终输出做一遍 image pass。

It feels dissatisfying for the people who want to just train models.

对那些只想训练模型的人来说,这感觉挺不尽兴的。

It's interesting, right?

挺有意思的,对吧?

Like it's it's also somewhat exciting like you brought up earlier, a lot of the gains don't come as much from the video.

而且这也有点令人兴奋,就像你前面提到的,很多提升并不那么多地来自视频本身。

Like I think you can see that in the language model space too right anthropic very very good at coding their multimodal not the best right they have basic input PDF but like you know there's clearly a disconnect in the quality of their image video processing audio processing yet intelligence very top tier other labs Gemini OpenAI xAI you can add modalities but it's not like they're unlocking crazy capabilities right so it's interesting yeah It's interesting to see that like the video models capability increase actually come from language model being more intelligent.

我觉得你在语言模型领域也能看到这一点,对吧,Anthropic 非常非常擅长编程,他们的多模态却不是最好的,对吧,他们有基础的 PDF 输入,但你知道,他们的图像视频处理、音频处理的质量明显有落差,然而智能却是最顶尖的;其他实验室 Gemini、OpenAI、xAI,你可以加上各种模态,但这并不意味着他们解锁了什么疯狂的能力,对吧,所以这挺有意思的,对,能看到视频模型能力的提升其实来自语言模型变得更聪明,这很有意思。

I think video agent like it it can unlock more stuff that might you might imagine.

我觉得视频 agent 能解锁比你想象中更多的东西。

So there there's a few things.

所以有这么几件事。

So one thing is when we are prompting these models so most of the people were actually not very good at prompting

其中一件事是,当我们给这些模型写 prompt 的时候,大多数人其实并不太会写 prompt

actually language models have a better sense of how to prompt AI models.

其实语言模型更懂怎么给AI模型写prompt。

AI models know AI models better.

AI模型更了解AI模型。

So if you jointly train these models maybe the model have a better sense of how to prompt each model like different different model might be different.

所以如果你把这些模型联合训练，模型可能会更清楚怎么给每个模型写prompt，因为不同的模型可能需要不同的prompt方式。

Another thing is it might not as simple as just a like a generate a few clips and slaps them together using ffmpeg like you might there might be more like image and video editing tool appear in this process.

还有一点是，它可能不会像「生成几段clip、再用ffmpeg拼在一起」那么简单，在这个过程里可能会出现更多类似图像和视频编辑的工具。

Say if you want to exactly add a add a blob of text at this time step.

比如说，如果你想在某个时间点精确地加上一段文字。

The videos model video models might not get that intention very precisely but these are possible using these deterministic tools.

视频模型可能没法非常精准地理解这个意图，但用这些确定性的工具就可以做到。

The long the video agents can use all all sorts of tools.

视频agent可以使用各种各样的工具。

So you don't have to put all of the capabilities into the transition model itself.

所以你不必把所有能力都塞进transition模型本身。

Yeah, I I think that's very true.

对，我觉得这非常对。

Um, no.

嗯，不。

So, so for what it's worth, I think you're right.

所以，说实话，我觉得你说得对。

I think that this will be a big category.

我觉得这会是一个很大的品类。

I think probably you are predicting like the next one year in video is going to be all this.

我觉得你大概是在预测，视频领域接下来这一年会全是这些东西。

Do you have a time time prediction for how when this stuff ramps up?

你对这类东西什么时候会起飞，有没有一个时间预测？

Like

比如说

I mean, they already started.

我是说，它们已经开始了。

Is you are we?

是你……是我们？

So, no, it's so it's so good.

所以，不，它真的太好了。

I think the last one's just longer.

我觉得最后那个只是更长一些。

Uh, it didn't give me a minute.

呃，它没给我整整一分钟。

You gave me 36 seconds, but you know, are we feeling it now?

你给了我36秒，但你知道，我们现在感受到那种势头了吗？

Is there going to be inflection?

会出现拐点吗？

Is there any timeline predictions you want to make?

你有没有什么时间线预测想说的？

I guess by the end of this year is this is going to be a big hit.

我猜到今年年底，这会成为一个大爆点。

So the the inflection point will be where and the the videos generated by video agents can get to like production great qualities.

所以那个拐点会出现在——视频agent生成的视频能达到接近production级别的质量的时候。

it it can be presented and it can be it can be distributed in in ads and when once that happen I think the enterprise will have much more budget for video models because the agents are inherently more expensive than than the than the video models themselves because they do this iterative process they generate many many variations S yeah but once once these models have this pass this usability threshold I think it's it's going to be a exponential growth beyond that.

它能被拿出来展示，也能被投放、分发到广告里，一旦那一天到来，我觉得企业会为视频模型投入多得多的预算，因为agent本身天然就比视频模型本身更贵，因为它们要做这种迭代的过程，会生成很多很多个变体，对，但是一旦这些模型跨过了这个可用性的门槛，我觉得在那之后就会是指数级增长。

Yeah, I would uh fund a company right now based on this uh listening.

对，听完这段我现在就愿意投一家公司。

Um so I think you're right.

嗯，所以我觉得你说得对。

One thing I'm I'm surprising I'm reflecting on the whole like past hour or so conversation.

有一件让我意外的事，我在回顾过去这一个小时左右的整段对话时想到的。

You I think you're into world models and video generation for video generation sake.

我觉得你是为了视频生成本身而投入到world model和视频生成里的。

I think there are a lot of other world models people.

我觉得还有很多其他做world model的人。

We've interviewed a lot of them.

我们采访过他们中的很多位。

Uh General Intuition and File and all those guys and and Moonream which I I I think I told you about.

呃，General Intuition、File还有那帮人，还有Moonream——我想我跟你提过的。

Uh Moon Lake.

呃，Moon Lake。

Moon Lake.

Moon Lake。

I keep saying Moonream.

我老是说成Moonream。

God damn it.

该死。

Moon Lake.

Moon Lake。

A lot of them actually say like robotics is the endgame like embodied robotics like you want real time, you want interactive, it is to interact with the physical world.

他们中很多人其实会说，机器人才是终局，比如具身机器人，你想要实时、想要交互、要和物理世界互动。

You're not that concerned about it.

而你对这件事没那么在意。

I think robotics will be a uh will be a big part of it for sure.

我觉得机器人肯定会是其中很重要的一部分。

I guess the the process might happen naturally.

我猜这个过程可能会自然而然地发生。

So, so my prediction on robotics is that the problem of physical AI might be solved like without actually need to

所以，我对机器人的预测是，physical AI这个问题也许可以被解决——而不需要真的

be in the real world

身处真实世界里

need to be in the real world.

身处真实世界里。

So it might it might get solved by a video LM is very strong video capability.

所以它可能会被一个视频LM非常强的视频能力所解决。

So, so remember we talked about the the real time interactive long horizon video once these models.

所以，还记得我们聊过的实时交互、长时程的视频吗，一旦这些模型……

So, so now these models are just training training on like screen recordings and computer screens.

所以，现在这些模型只是在类似屏幕录制和电脑屏幕这样的数据上训练。

Once these models can use computers and understand the future state of computer extremely well, the robots might be might be one of the one of the tools like a very powerful AI can can use.

一旦这些模型能够使用电脑、并且极其准确地理解电脑的未来状态，那么机器人可能就只是一个非常强大的AI可以使用的工具之一。

So the powerful AI might just uh be able to control the physical embodiment naturally.

所以那个强大的AI也许就能自然而然地控制物理上的具身形态。

I see that for sure.

这个我完全认同。

Cool.

酷。

I I I know I know we are coming up on time.

我知道我们时间快到了。

uh you had you left one more spicy topic which is why you left xAI.

呃，你还留了一个更劲爆的话题，就是你为什么离开了xAI。

For me um there's there's a lot of a lot of research you want to do that you cannot do at as a company and also like the the priorities and and objective the company typically can can change very fast.

对我来说，嗯，有很多你想做的研究，是你在一家公司里做不了的，而且公司的优先级和目标通常变化得非常快。

It is it's also the same for xAI.

xAI也一样。

So, so now is kind of like the the time to there are some research I I want to do especially more on language model side like I I cannot do at xAI.

所以，现在算是一个时机——有一些研究我想做，尤其是更偏语言模型这一侧的，在xAI我做不了。

Oh, uh okay.

哦，呃，好。

Yeah, cuz you're you're basically leaving you you you had this whole transition from computer vision to world models, video generation uh to now now you're like focusing on LMS

对，因为你基本上是要离开了，你经历了这整个从computer vision到world model、视频生成的转变，呃，到现在你要开始专注在LM上了。

but it seems like you know a lot of you saying focusing on LMS you really in the past hour described how it all ties together, right?

但是感觉……你说要专注在LM上，可你在过去这一个小时里其实描述了这一切是怎么串在一起的，对吧？

like

比如说

but I don't know what what do you mean by focusing on all is there

但我不知道你说的「专注在这一切上」是什么意思，是不是有

I realized the fact that the the video models even like in the beginning the game might come from improvement on diffusion technology but this is a point where actually most of the gain come from the language models themselves

我意识到一个事实：视频模型，即便一开始的进步可能来自diffusion技术的改进，但到了现在这个阶段，其实大部分的收益都来自语言模型本身

is a huge black pill for anyone who has like spent a career in like generative uh media

对任何在generative media领域投入了整个职业生涯的人来说，这是一颗巨大的black pill

I mean that's an extreme view right there that you still definitely need a bit of both, right?

我是说，那是一个相当极端的看法，你肯定还是需要两者都有一点的，对吧？

There's just uh it seems like more pressing impactful work to do now on language model side.

只是……现在在语言模型这一侧，似乎有更紧迫、更有影响力的工作要做。

Do you have any similar predictions you so you predict the video agents?

你还有什么类似的预测吗?你之前预测了 video agents。

I think you will be right on the language side.

我觉得在语言这一侧你会说对。

What are you looking for in the next one year?

在未来一年里,你在期待看到什么?

I think one thing pretty pretty interesting I I think might be happening soon is the the language models will be like context aware and manage its own context.

我觉得有一件挺有意思的事,可能很快就会发生,那就是语言模型会变得能感知自己的 context,并管理自己的 context。

Yeah.

对。

So some like from from the video model side we we've been suffering from the long horizon issue like we want to generate video longer and longer and we've been trying to solve the context length issues through various ways one one thing is just brute forcing train longer context length another is to manage the context better I think the same thing in language model is also going to be happening soon.

从 video 模型这一侧来说,我们一直在被 long horizon 的问题所困扰,比如我们想生成越来越长的视频,我们一直在通过各种方式来解决 context length 的问题,一种是直接暴力地去训练更长的 context length,另一种是把 context 管理得更好,我觉得同样的事情在语言模型里也很快会发生。

So for example like the longish models they they're not aware of how long their own context length is once they hit like 80% or something the automatic context comparison is getting triggered and and the model is not aware of that when it's working and some maybe it's good for the models to to know oh I'm I'm approaching like 80% or something and something also pretty interesting like for example in open claw or like you every time you type in something like uh at times uh the current local time is automatically attached to your message.

比如说,那些较长的模型,它们并不知道自己的 context length 有多长,一旦它们撞到比如 80% 之类的阈值,自动的 context 压缩就被触发了,而模型在工作的时候是意识不到这一点的,或许让模型知道"哦,我快到 80% 了"之类的会是件好事;还有一件也挺有意思,比如在 open claw 里,你每次输入什么东西的时候,呃,当前的本地时间会自动附加到你的消息里。

So the model actually know what time is it.

所以模型实际上是知道现在几点的。

So this is making the the model time aware and also like in two calling the a lot of the intermediate to call results automatically prune.

所以这让模型有了时间感知,还有比如在 tool calling 里,很多中间的 tool call 结果会被自动裁剪掉。

So there's like context removal, context addition and context compaction.

所以这里就有 context removal(移除)、context addition(添加)和 context compaction(压缩)。

So all of these are from the harnesses themselves but from our experience the heristic engineering all the models get us absorbed into the models themselves I guess that's something very interesting to explore so infinite context

所有这些现在都来自 harness 本身,但从我们的经验来看,这些 heuristic engineering(启发式工程)最后都会被吸收进模型本身,我觉得这是一个非常值得探索的方向,也就是 infinite context(无限上下文)。

maybe no but it's it's interesting right um

也许吧,不过这确实挺有意思的,对吧,嗯。

it is in the space of memory and continual learning and

这属于 memory 和 continual learning(持续学习)的范畴。

I don't know it's also like in the space of agent harness use right you're saying

我不确定,不过它其实也属于 agent harness 使用的范畴,对吧,你的意思是。

he's saying doesn't want to do it in a harness, right?

他的意思是不想在 harness 里做这件事,对吧?

No, no, but models are also being trained on unit using harnesses, right?

不不,但模型本身也是在使用 harness 的过程中被训练出来的,对吧?

So, some of it is you could say implicitly leaking in, right?

所以其中一部分你可以说是隐式地渗透进去的,对吧?

Um, you know, part of that post training of language models is okay using it in coding harnesses in which case, you know, when are sub agents spawned?

嗯,你知道,语言模型 post training 的一部分,就是在 coding harness 里使用它,在这种情况下,你知道,sub agents 什么时候被 spawn 出来?

When is convection going to happen?

什么时候会发生 compaction(压缩)?

uh it's not explicit like you know you have this much token window which I don't know if you want it to be is that'll change but it's it's somewhat leaking in there.

呃,它并不像是那种明确的"你有这么大的 token window",我也不知道你希不希望它是这样,这个会变的,但它就是多多少少渗透进去了。

I'm imagining what if the model have access to the whole the code of the agent harness itself and be able to modify it whatever you want.

我在设想,如果模型能访问 agent harness 本身的全部代码,并且能随意修改它,会怎么样。

Say if the agent harness is short enough, you can just put in the context length in the system prompt and then the model say when I want to spawn a future version of myself, I can modify the agent harness.

比如说,如果 agent harness 足够短,你就可以直接把它的 context length 放进 system prompt 里,然后模型说,当我想 spawn 一个未来版本的我自己时,我可以修改这个 agent harness。

For example, if I the Asian harness can be or when I'm reading a long document, I can choose to read the whole thing in chunks and come back, smash the summary together or I I can just read the first uh first 200 lines and discard the rest and all all kinds of choices if they can be made by the model themselves.

比如说,如果这个 agent harness 可以做到,或者当我在读一篇很长的文档时,我可以选择把整篇分成一块块来读,再回过头把这些摘要拼到一起,或者我也可以只读前面呃前 200 行,把剩下的丢掉,所有这些选择都可以由模型自己来做。

It might be very interesting to to see that the model can like a program the model can program itself online in test time.

能看到模型可以像编程一样,在 test time(测试时)在线地给自己编程,那会非常有意思。

Yeah.

对。

Uh so the self modifying harness is also part of uh open claw and pi but I I think there's a lot more work to do there.

呃,所以这种自我修改的 harness 也是 open claw 和 pi 的一部分,但我觉得那里还有很多工作要做。

Very cool.

非常酷。

I think part of me it's kind of curious you know like I think you are part of big lab right and there's this career path of a researcher at a big lab which is you are you train models you get more compute you train better models you keep going and somewhat I feel like you're opting out of that and if I were you I'll be like oh I think this is like a bit of a career risk you know what I mean

我心里有一部分挺好奇的,你知道,我觉得你是身处一个大 lab 的,而在大 lab 里有一条研究员的职业路径,就是你训练模型、拿到更多算力、训练出更好的模型、一直这样走下去,而我感觉你多少是在主动退出那条路,如果我是你,我会觉得"哦,这算是有点职业风险的",你懂我的意思吧。

I I don't have any comment apart from like you're very strongly convicted I think that a lot of people in your shoes would not be doing what you did.

除了说你信念非常坚定之外,我没什么好补充的,我觉得很多处在你这个位置的人不会做你所做的事。

Yeah.

对。

Speaking of my career, if I I look back actually there there were there were a lot of huge transitions.

说到我的职业生涯,如果我回头看,其实里面有过很多次巨大的转变。

So So 10 years ago, I was I was doing research with the restnet authors Shan John and Jensen.

10 年前,我在和 ResNet 的作者们,Shan、John 和 Jensen 一起做研究。

Yeah.

对。

At that time the the research were completely different.

那个时候的研究完全是另一回事。

It was like uh mostly computation like image recognition, object detection, object tracking.

那时候主要是计算机视觉方面的,比如图像识别、目标检测、目标追踪。

I I was also doing neuron net compression at that time.

我那时候也在做神经网络压缩。

It was quite different from knowledge dissolution these days.

这和如今的 knowledge distillation(知识蒸馏)相当不同。

And at that time I I was I wanted to be a professor and I I applied when I applied for PhD I already had a few first author papers at top conferences.

那时候我想成为一名教授,我在申请 PhD 的时候,已经有几篇在顶会的第一作者论文了。

So I I confidently applied the the top schools but turns on I got rejected by all of the top PhD programs.

所以我很自信地申请了那些顶尖学校,但结果我被所有顶尖的 PhD 项目都拒了。

So I had to I had to go to the industry at that time.

所以那时候我只好去了工业界。

I was at Facebook hair research fair led by

我当时在 Facebook 的 FAIR 研究院,领导它的是……

I want to talk about VJA but it's different.

我想聊聊 VJEPA,不过那是另一回事了。

Anyways yeah we can leave it for another time.

总之,对,这个我们可以留到下次再聊。

Uh yeah switched to at that time I switched to self-surprise learning.

呃,对,那时候我转去做 self-supervised learning(自监督学习)了。

It was it was quite different from what I was doing in contribution.

这和我之前在计算机视觉里做的东西相当不同。

Yeah.

对。

And and after that it's NVIDIA Cosmos.

在那之后就是 NVIDIA Cosmos 了。

So I realized scaling up was extremely important.

我意识到 scaling up(规模化)极其重要。

So at NVIDIA I I was mainly focusing on scaling.

所以在 NVIDIA,我主要专注于 scaling。

So one thing is cosma scaling the video description models to to a few billion parameters and another thing is uh I was working ones the the megatrones was the first was was the first framework open source to be able to train these at very large scales like 100 billions parameters to even trillions parameters efficiently at like 40% MFU

一件事是在 Cosmos 把 video 描述模型扩展到几十亿参数,另一件事是,呃,我参与了 Megatron,它是第一个能够在超大规模上高效训练这些模型的开源框架,比如把 1000 亿参数、甚至上万亿参数的模型以大约 40% 的 MFU 训练出来。

[sighs and gasps]

[叹气并倒吸一口气]

and going to switching to xAI was trying to work on even larger compute scale even further and yeah looking at this trajectory I actually work worked on a lot of different things so I feel actually within within ML it's actually easier to switch And then you think like a lot of people might have manifested that oh I work on I work on computer vision I always have to work on computer vision and I cannot switch to language and but but from my experience at least at at NVIDIA I worked on both language modeles and also video models it's it's actually not the case a lot of a lot of the core principles how who train large models are are largely the same.

然后转去 xAI,是想在更大的算力规模上工作,把这条路再往前推,是的,看着这条轨迹,我其实做过很多不同的东西,所以我感觉,在 ML 领域里其实是更容易转换方向的,你会觉得很多人可能会给自己设限,"哦,我做的是 computer vision,我就永远得做 computer vision,我没法转去做语言",但根据我的经验,至少在 NVIDIA 我既做过语言模型也做过 video 模型,其实并不是这样,很多核心原理,也就是怎么训练大模型,在很大程度上是相通的。

And yeah, for for me I feel right now the the bottleneck uh for for video models is actually the language part the the agent which which is why I want to go to work more on all one thing is it's it's a bit of a challenge.

而且,对我来说,我感觉现在 video 模型的瓶颈其实是语言的那部分,也就是 agent,这也是为什么我想多去做这方面,一方面这确实是个不小的挑战。

I don't think it's a huge uh jump.

我不觉得这是个巨大的跨越。

So

所以……

yeah I mean Kudos to you.

对,我是说,向你致敬。

I think you have a lot of uh strong vision there.

我觉得你在这方面有很强的 vision(远见)。

Yeah, I think that was mostly everything that we wanted to cover.

对,我觉得我们想聊的基本都聊到了。

You've been very generous with your time and I I you know it's really nice that you are able to share all these things now.

你在时间上非常慷慨,而且你知道,你现在能分享这么多东西真的很棒。

We don't have to go through xAI to to clear everything.

我们不用非得通过 xAI 去逐条审核清楚。

Uh but also we I think we we we didn't get you in trouble.

呃,不过我觉得我们也没有给你惹上麻烦。

It's a lot of good stuff about xAI compared to what you just see in the releases, right?

跟你只在发布内容里看到的东西相比,这里面有很多关于 xAI 的好料,对吧?

You don't realize how many more levels there are to it.

你意识不到它背后还有多少层深度。

xAI, please do more podcasts.

xAI,请多做点播客吧。

Uh anyway yeah but thank thank you for uh sharing.

呃,总之,对,谢谢你的分享。

It's been very kind and and also like I want to hear more from you.

你非常友善,而且我也想多听你讲讲。

I think you are going to embark on your your next phase.

我觉得你即将踏上你的下一个阶段。

You haven't announced what you're doing next but clearly you have you know uh more vision and more ambition on on this path and I think you're you're basically kind of gradient descenting to like whatever your final form is.

你还没有公布你接下来要做什么,但显然你在这条路上有着更多的 vision 和更大的野心,我觉得你基本上是在用 gradient descent(梯度下降)一步步逼近你最终的形态。

Thank you.

谢谢。

Yeah.

对。

Yeah.

对。

I I'll share more about my next chapter soon.

我很快会分享更多关于我下一个篇章的事。

Thank you for having me.

谢谢你们邀请我。

Thanks for coming.

谢谢你能来。