How Fireworks AI Scaled 100x Traffic Without Exploding GPU Costs

Most AI startups struggle to scale their infrastructure as they grow. But what if you could handle 100x more traffic in just six months without breaking the bank on GPU costs?

Lin Qiao, CEO of Fireworks AI, knows this challenge intimately. After spending years building AI infrastructure for Meta's billions of users, she watched countless companies struggle through their AI-first transition without the right tools, team, or hardware. Her solution? A platform that processes over 150 billion tokens daily while helping businesses minimize their GPU costs.

In this interview, Lin reveals how Fireworks AI achieved 100x traffic growth in six months, why startups must think in 10x leaps rather than incremental improvements, and her contrarian hiring philosophy that values aptitude over experience. She also shares her vision for compound AI systems and the critical mindset shift that separates successful AI startups from the rest.

Lin: Hi, I'm Lin. I'm CEO and co-founder of Fireworks AI. I started Fireworks AI with my co-founders late 2022. We have a long time experience working at Meta, building AI infrastructure from ground up.

We work with many companies in the industry and experience their pain going through this AI first transition without a team like us, without the know-hows, without proper hardware, without the right sovereign tools. Today we're processing more than 150 billion tokens per day and generating more than 1 million images per day.

From the funding point of view, we have raised first round from Benchmark, 25 million and then recently, we just closed a round led by Sequoia with 52 million investment and post money 552 as a valuation. So our evaluation grew by 4 times.

Lin: Throughout my career, I've been through waves of technology enabled business transformation, waves of it. And that particular intersection of technology and business deeply interests me. And then we go through mobile first transition. That's a huge tectonic shift. I want to join a company where they are on the forefront of this transition, that's Facebook.

When I joined Facebook just finished the transition from desktop to mobile first. So when I initially joined Meta, I joined the data infrastructure team. And when I'm running the team, I look at the stats of data growth. And the data growth is mind blowing. It's just so fast, it doesn't make sense. I asked myself, I need to figure out what's going on here.

And I did a drill down and figure out, oh, majority of data growth is driven by AI. That is clear to me. That's the future. And there's a lot of unsolved problem because it's a new emerging area. So I moved to the AI space, and that's why I start to build a team from 5 people to 300 people over the course of 5 years.

Lin: Me and my co-founder have spent years at Meta, and we have hundreds of people building all the AI infrastructure, supporting Meta's AI first transition. Well, I also have many friends working other big companies in the industry. They are way behind me and when they go through the AI first transition, they don't have hundreds of people, machine learning team or AI infra team to help them.

So our mission of starting Fireworks AI is to enable new business to flourish, building on top of this innovative generative AI technology without 100 people, machine learning, engineering team, and infrastructure team.

Generative AI is very big. The model is very large. It's just very slow to run these models. So not having very low latency and fast response makes a product of not appealing at all. So we know this is a big pain point.

Another big pain point is the generative AI model is so big, and they have to run on GPU and they have to acquire a lot of GPU and GPU is so expensive. If they're lucky to get the GPU, they have never seen such a big bill before. So we design our software stack to minimize your GPU needs to solve your problem. In that way, a signal control the cost, operation cost for the company. So then our customer will have a viable business when they scale quickly to 10, 100, 1,000x more customers.

Lin: Within the past half a year, our traffic grow by 100 times. Today we are processing more than 150 billion tokens per day and generating more than 1 million images per day.

There are a few key things about startups. One is do not work on incremental work. Do not pick on incremental things. It's human nature to work on projects that we know we can deliver, but it's not for startups. Startups are pushing for 10x. 10x faster customer adoption, 10x faster infrastructure latency, 10x higher scale, startups are only here for 10x. It's for a huge leap.

The second is startups are not for slow moving pace. It has to kind of outrun and outpace competition. There should have already been incumbents occupying the space and the speed is advantage of startup because we don't have the burden of coordination. We don't have the burden of slow decision making.

Lin: Be able to say no is essential for moving fast because over time, you can justify, hey, why we should add this, why we should add that. And then the same person got time slices into multiple different top priority things. So I think laser focus on prioritization and really ask hard questions. Does this project really move the bottom line of our business? Can we visually see that our metrics will change?

I think that have the principle to drive that conversation and be able to say no, it's very important. Again, the goal is not to make people happy. That's not the goal. The goal is to make a solid product decision or strategy decision and have everyone laser focus on delivering that. That's the key point.

Starting this company, we're constantly asking ourselves, are we working on the right problems? Can we deliver this today? And can we move faster? I think that mentality is deeply grind into us. And that's also essential to build a successful startup is kind of have a huge sense of urgency to keep asking, why not today? Why not yesterday? Why not faster?

Lin: 2023 has been a year where we laser focus on a large language model with text in, text out. Of this year and beyond, that will not be sufficient because a lot of business tasks will emulate what's happening in real life. In real life, we communicate way beyond text. We speak and we use visual to collect signals.

So that's where we have to expand beyond large language model into models who understand images, who can generate images, who understand audio, who can generate audio, and I firmly believe the direction for the industry to move forward is into multimodality. And today, on our platform, we already serve more than 100 models across all these modality from large language model to audio models to image generation models. We have a very broad variety of modality for our customer to pick and choose from.

Lin: With that said, that's still not enough, because every single model has limited knowledge. And the fundamental reason is those models are like child in the school, they learn from textbook, and those gen AI models learn from training data. And training data is finite, it's not infinite. If you ask the question, it will have to give you an answer in a probabilistic way. When it goes out of its knowledge, it's going to hallucinate.

So that hallucination is a big problem for application developers. And the way to address that is to let each model focus on its specialty. Only answer questions when it's good at. And we are building a power routing layer. It's called function calling. So we have our proprietary knowledge to figure out which specialty model to routing, to give the best answer.

It does not just route into different models, different modality. It can also route into APIs. APIs could be search, could be weather, could be stock price, could be many other things. So, then this model is able to pull together the totality of knowledge to give the best answer to our users. This is the form and shape of compound AI system. And Fireworks is growing into and building into our next generation compound AI systems.

Lin: Very simple aptitude. It's not about experience. I actually prefer aptitude over experience. I need to see the fire in the belly, super hungry, super motivated. That trumps anything else. Because this is a fast moving technology. Everything's new, how fast you pick up, how fast you're a learner, how determined your problem solver. That makes a huge difference.

Lin: In the early stage of my career, I have this deep imposter syndrome. So I think a lot. It's actually a way of self reflection. And I constantly think about what I can do better, how I can do things differently. But later on becomes a habit. It just, when we take some action, we do things, I will observe, hey, is this going as I expected? And if it's different, then why it's different and how we can be more efficient?

And what are the small changes or big changes we can introduce to the process, to the way we do things to be better, to move faster. So and then I share my thoughts with my team. Sometimes I will ask them, hey, what do you think? How this is going, what we can do differently. So just kind of build this muscle over time. We just think together as a team, to kind of always there's always room to improve.