From $200K to $30M: How Seoul's Cyber Warriors Built Video AI

When venture capitalists asked Twelve Labs CEO Jae Lee how long it would take to index a billion hours of TikTok videos, he froze. His team had been bragging about handling a million hours – which would take them 10 years to process at scale.

That moment of reckoning came during a 3:30 AM pitch call from Seoul, where Lee and his co-founders – fresh out of Korean military cyber command – were trying to convince Silicon Valley investors to bet on their wild idea: building AI that understands video like humans do. Today, Twelve Labs has raised $30M and counts NVIDIA, Snowflake, and over 20,000 developers as customers.

In this interview, Lee shares the brutal lessons of building video foundation models from a bagel shop outside military barracks, why saying "no" to respected mentors saved his company from becoming "an underwear company," and how winning a computer vision competition with their last $200K changed everything.

Jae Lee: Hi, my name is Jay. I'm one of the co-founders and CEO of 12 Labs. 12 Labs is an AI research and product company based here in San Francisco and Seoul. We're building video foundation models for developers and enterprises building video centric products.

We basically build humongous AI models that can understand videos like humans, and we serve it to developers via APIs that are looking into building really powerful semantic search or classification or summarization into their products. We currently have a little over 20,000 developers that are actively using our search API.

So I was born in Seoul. I spent about 10 years in Seoul. I had a chance to move to the states when I was 11, so went through elementary and middle school in Knoxville, Tennessee. I was very interested in expanding my perspective and exploring the new world.

My first experience with software engineering was MATLAB. My uncle was actually getting his PhD at the University of Tennessee, so I see him plotting distribution graphs and things like that, which made me curious what he was doing. That's how I got into playing around with small data sets and doing the same thing that he was doing because I wanted to be relevant and I wanted to talk to him.

It sparked my interest in learning more about how do we capture all this data and be able to create a system that can really understand the distributions of the things of the world. It felt like if you had the understanding of that, it gives you power to predict anything. I went to Berkeley for college and studied computer science, so I really geeked out on AI and software engineering.

Jae Lee: I was drafted into this organization called Korean Cyber Command where like-minded people were already there with incredible knowledge in software engineering and AI. I joke about it - I served the country with keyboards rather than a rifle.

I was really fortunate to have met my chief architect SJ and then Aiden joined in. I still clearly remember Aiden with his buzz cut coming in from boot camp, but from day one, we knew we had this common interest in AI and what we can do as young scientists to really push the frontier of AI development. We spent a lot of time reading papers, discussing, arguing.

It turned out there were two clear paths. One was after military we go pursue a career in academia, become professors, or start something of our own. Looking back, we realized we were spending so much time together and military is this really special setting where you're basically jamming 50 to 100 twenty-year-olds with testosterone.

We thought if we're having this much fun in the military, imagine what we can do when we go out. So it was pretty clear to us that we're going to start something.

We spent about a year and a half really thinking about what is the next frontier for AI and how can we contribute to pushing that boundary. There's this seminal paper called 'Attention is All You Need,' which some people call the Transformer paper, which is making a lot of impact now.

For text and image based foundation models, probably capital was going to be a moat - whoever raises the most amount of money. What we realized was there's still a lot of unexplored research areas for multimodal video understanding. Rather than capital, it's probably going to be really passionate, smart, but also slightly dumb people that's dumb enough to start it. We have really good chance of succeeding.

We realized that with the explosion of video and other complex multimedia data, it's going to be the infrastructural data for the internet. Developers and enterprises need something better than object detection or transcription to make sense of all this video data that humanity is creating. So it was a no-brainer to start building for video understanding.

Jae Lee: We didn't all join Korean Cyber Command at the same time. SJ was already 6 months ahead of me and Aiden was 6 months behind. We decided we're gonna start this company but then SJ is leaving next year and then I'm leaving the year after and then Aiden's 6 months after. So how do we do it? It was genuinely very scary.

I remember SJ was discharged on Thursday and he came back to the military base Saturday of that week with our laptops. He took us out in front of our military base - there was a bagel shop called Last Bagel and that was like our office where SJ would bring all of our laptops and we would do our research and prototyping.

We did that for 6 months and then I got discharged and then I did the whole laptop carrying and taking out Aiden. So we did that for like a good year until everyone was out.

We had a bunch of friends that were working in AI and crypto and blockchain and Web3 was just booming. We had a mutual friend that had a really nice office in Seoul, and it was like, 'Oh you guys can come in and use our office space.' That's what we did. And then after 3 weeks that company went bankrupt. Really scary - people started coming in and we had our desktops and GPUs all set up there and we got really scared so we brought everything back out.

We found a really tiny office, probably like the size of a dressing room. That's where all 5 of us spent the next 6 months before we raised our proper seed round.

Looking back, if we were to do it again, I don't know if we'll be able to do it. Some people say ignorance is bliss, and I think we were just really naive and really excited about building this company. Not knowing what was ahead allowed us to kind of do what we did.

Jae Lee: The founders were pretty much broke because we spent 2 years in military and I think we had $2000 to start with, so barely enough to do anything. We needed to figure out what was going to be impactful that we can do given our current resources that'll put us on the map.

Our tactic was we're gonna talk to a bunch of customers, and there were early believers in 12 Labs who took our APIs and built awesome things with us, but we needed more exposure. As a team we decided to participate in ICCV - International Conference in Computer Vision. They were putting on this awesome competition for video understanding.

We talked to Aiden - we have nothing to lose and only to gain. The team was extremely supportive of Aiden spearheading that effort. We needed compute and we needed determination to put some serious cash behind it. Back then for 12 Labs, $200,000 in compute was a lot of money for us. Just thinking that we're gonna blow through $200,000 in 10 days in compute was really scary, but the team was able to use that precious capital and build something incredible that helped us win the competition.

I think the important thing is if you're building something really impactful and you think it's going to significantly change the industry that you're in, there will always be someone that has a very similar thesis. It's just a matter of how do you get yourself out there? How do you let people know that you exist? For us that was the competition.

After winning the competition, companies like Index Ventures and Radical Ventures had very strong thesis around multimodal AI. These amazing companies actually came inbound - they reached out to us and we started conversations. The conversations turned into next conversations and we talked about technology, and it just happened very serendipitously.

Jae Lee: The first pitch deck - I was in Seoul and my first call with Index Ventures was at 3:30 AM Seoul time and we didn't have a pitch deck. We just felt like this is our first meeting and we knew nothing about fundraising then. We didn't even know this was going to be a friendly introduction, but I just felt the need to build one. So I remember staying up till 3 AM building it.

The storyline was quite simple. We didn't have much to show for it. The idea was: the problem that we're solving is massive. 80% of the world's data is in video and there's no adequate solution out there for developers and enterprises to make sense of it all. That is the market that we're tackling. We want to index all of that 80% of the world's data, and this is the underlying research work that we've done.

VCs asked a lot of hard questions. The most memorable one: if I bring you TikTok as a customer and they want to index a billion hours of content, how long does it take? I think we were thinking about maybe a million hours is going to take 10 years. That's when we realized we should never be comfortable with what we've built. There's this whole new incredibly large world out there.

Maybe some people are impressed with our system being able to index a million hours, but there's others that are thinking about a billion, 10 billion, 100 billion hours. That was a really challenging question because we said during our pitches we wanna index all of the world's videos and that question kind of made me stunned. I thought about all of the technical issues that we had at the time.

So Young (Co-founder): We had a customer who was paying for our product, but they weren't actually using it. We had gone through a lot of work to actually get them as a customer through sales and relationship building, but I think they were extremely early and we had almost pushed the sales to happen.

What we learned from that experience was we have to optimize for the right things. Even early on, it might have been better for us to not actually make the sale because the customer probably wasn't ready. They didn't have the passion or the innovative drive that our other customers had, but we were optimizing for hearing the yes.

We tried so hard to make that no into a yes, and we had succeeded, but at the end, I think it turned out that we probably should have kept it at a no and focused on all the other customers where the yes was more clear. For earlier products and technologies where resources are limited and you want to build for your best customers and the innovators in every field, you should probably start to optimize for hearing the no than the yes because that will help you find the right direction faster.

The types of companies that we need as early customers are true innovators in their field, whether they come from the content creation space, law enforcement space, e-learning, and so on. If you find the right customer who's innovative, you don't need to explain anything to them. You show them a use case-based demo - for us it would be we would index videos that resemble the customer's and then we show them how you can search or generate text very easily like a person would, having watched the video. They can draw out the full map of what they want to provide to their customers or users.

Jae Lee: I think we make stupid mistakes probably every day. The one that I regret the most is we knew we had this conviction around building a foundation model, but then we didn't have any data point as to how do you build that company. I think I blindly believed that startup mantra like identifying a narrow problem and building a very narrow solution for it.

We spent a lot of early days thinking about if we have this really powerful AI that can understand videos, what do we do with it? We knew from the get-go that we want to serve it to developers and enterprises, but we had mentors and other founders saying 'oh you should build TikTok 2.0 or you should build YouTube 2.0.' We spent a lot of time thinking maybe TikTok 2.0 makes sense or maybe Gong 2.0 like sales call analysis, but that didn't really excite us because we knew we're good at building the infrastructure and helping developers build the next thing.

That's probably the stupidest thing that we've done - spending time on thinking about things that we're not excited about. As a founder and CEO it's really hard to not get distracted. If you're a first-time founder or a young founder, your mentor's advice means a lot to you, but having your own grounding, relying a little bit on your own gut feeling is very important.

What we realized is if you try to accommodate all of the advice that you get from your mentors, your company will most likely become like an underwear company, totally different from what you wanted to build. So my key takeaway is having some fundamental foundation for yourself and for the company and be able to say thank you for your advice but no thank you. Having the gut to say no to someone that you respect - as a founder, I think I grew a lot.

Jae Lee: 12 Labs has a multi-year compute partnership with Oracle Cloud Infrastructure where we get all the state-of-the-art media. Oracle had put together this small event for their key partners that are building foundation models and Aiden and I had a chance to meet with Jensen because Jensen was at that event.

We had, I think, 5 to 10 minutes to talk about 12 Labs and it seems like he has a special place in his heart for computer vision and video understanding - that was one of the first use cases that NVIDIA chips powered. So we got to meet with NVIDIA folks from that event and then 12 Labs was featured in NVIDIA's 2023 GTC.

That sparked other people from NVIDIA to be interested in 12 Labs and the NVIDIA venture team reached out to us. It was quite casual. We were talking about 12 Labs and the future that we're drawing and the future of multimodal video understanding. The venture team also had an idea of how NVIDIA and 12 Labs can partner up more than just financial investment, but also think about really robust product partnership.

From then on, what 12 Labs is doing and what NVIDIA wants in vision and video understanding was just a perfect match. It happened quite naturally from conversing about our technology, our roadmap and NVIDIA's future in terms of producing really powerful chips for edge devices for smart cities. There's that natural fit of two companies' products really creating synergy.

Jae Lee: 12 Labs vision in the next 2 years is really becoming horizontal video understanding infrastructure for all of the businesses and developers that are working with video data. We want to enter into streaming as well, so real-time video data and really become a visual cortex for modern video applications.

Nowadays I am focusing mostly on hiring. 12 Labs is a group of great people. I spend a lot of time meeting great people. I want to be able to recognize greatness when he or she comes in. Good engineers or even just good people in general have these core values. I would go near their places and get together at a cafe and we would speak for 3-4 hours.

My way of deciding whether this person is a good fit for 12 Labs is whether I am able to learn from their core values. Everyone's really good at coding nowadays, but great engineers can apply their core values and their skill sets and is able to talk about the company that they're excited about and how they want to impact it. How do you see the product evolving? How do you see our interfaces evolving?

The best engineers are not necessarily the best coders, but having that perspective, really strong perspective and groundedness is very important and I try to look for that.