10 Paying Customers in 48 Hours: How Humanloop Found AI's Missing Piece

After 2 days, we had 10 paying customers. We had never felt pull like that before.

When Raza Habib and Jordan Burgess launched their AI startup Humanloop in October 2022, they discovered something most founders only dream of: product-market fit so strong that customers were literally throwing money at them within 48 hours of their pivot experiment.

In this candid interview, the co-founders reveal how they grew 60x in usage, doubled revenue last quarter, and raised $7M from top-tier investors by solving one of AI's biggest challenges: making large language models reliable and predictable for product teams.

Raza: Hi, my name's Raza Habib. I'm one of the co-founders and CEO of Human Loop. Human Loop helps companies that want to build AI products with large language models to both develop and then make those products reliable. Unlike software engineering, when you're dealing with deterministic code and you can predict how it will behave, using AI is non-deterministic. It will produce different outputs depending on very minor changes or other artifacts.

Typically the people using Human Loop are product teams who are building an AI focused application, collaborating with an engineering team to get it into production. So we launched the current version of the product in October 2022. We've grown, I think, about a factor of 60 in usage since then. We doubled our revenue in the last quarter and we've raised about $7 million of venture capital now from some of the leading investors in the world, including Index Ventures and Y Combinator.

Actually as an undergrad I studied physics. I was just really curious about it. I've always been a little bit nerdy. And my first job after university, I actually joined a small venture-backed startup called Just Park. It was a startup that was doing peer to peer car parking rentals. And I joined that company when there were only 5 or 6 people. By the time I left it was 45 people, it had grown to a million users. And so that was a really exciting experience to scale a company from a very small stage.

Raza: But it wasn't as deeply technical. Coming from a physics background, I wanted to work on something on the frontier of what was possible. And I also became increasingly convinced that AI was likely to be one of the biggest, if not the biggest, technological achievements that humans did ever. So this was probably around 2015. I decided to go back to school and I actually did a master's and then a PhD in machine learning.

So back in 2015, deep learning had sort of begun to take off. Today, you know, we use chat GPT, you have a conversation with it, it's incredibly fluent, it feels like you're talking to a person. In 2015, people were getting excited if you could produce a language model that could open and close a bracket, and things that now people take completely for granted, even two years ago felt like science fiction.

I mean, as soon as you start learning about machine learning, it's hard not to be fascinated by it, right? It touches so many different things simultaneously, like how can I build a machine that can learn in the same way that the human mind learns. There's an interesting technological question. There's a question about how human intelligence works, right? You just want to understand how might the brain work, so there's a biological question that's kind of fascinating, and then there's philosophical questions as well. Is it possible to build a machine, you know, in principle that can do all the things that a mind does? Will they be conscious, will they be sentient? So it's a fascinating field in and of itself.

Raza: As someone in physics, like it feels like the most exciting time in physics was the start of the 20th century when quantum mechanics was being developed. But I feel like the people who were doing that then, if they were alive today, would likely be working on AI. I feel like it's the most interesting intellectual problem of our time.

And when I finished the PhD, I actually spent a little bit of time working at Google AI, but I wanted to be working in an environment where the connection to product was a lot tighter and be part of a team, you know, a very small team with tight deadlines pushing towards something that feels like you can achieve something amazing, rather than the more relaxed environment of a bigger company. And I had really concluded that I wanted to start a company. I had a list of the ideas I was most excited about and the people I was most excited about. And I was trying to go through the smartest people I knew. Jordan was very high up my list. Jordan has an incredible eye for detail and taste in product. He's the kind of person who notices what makes the difference between a really good product and not.

Jordan: My name's Jordan Burgess. I'm the co-founder and chief product officer at Human Loop. Always been interested in technology and I had a lucky break at 15 years old that someone bought one of my websites from me. It was Wikipedia, but for guitar tabs and lyrics. I had my first exit for 4 figures of pounds. It's the Steve Jobs moment of like the world is created by people are no smarter than you, like you've created something and people have created value for it and paid you for it. That's a transformative moment in your life.

I was very fortunate that I could go on the exchange to MIT for my 3rd year and I just saw this ecosystem of startups, of people building stuff and got very enthused by that as a path for me. When I left university, I did start something then in the medical tourism space. If you don't have passion for that area, that can last 10 years, then you're not gonna really sustain the efforts needed to go and create that company.

My journey then was to figure out, well, what are the most impactful technologies that I should be working on and what can I make the investment in. So feeling around, understanding the space, and realizing that artificial intelligence is gonna be the most impactful technology in our lifetimes. So I decided to study in machine learning with the intention that ultimately this would be where I could happily spend a career working in.

Raza: So I remember going to visit Jordan in Cambridge, then we started to get into rooms together, and there's a room at UCL where all the four walls are whiteboards. So floor to ceiling, and I remember we covered every wall with different iterations of an idea. People talk about the moment you have an idea in a startup, but I much prefer the analogy of an idea maze, because once you say, OK, we're gonna build tools for NLP, OK, what does that look like? There's 1,001 decisions you have to make about who exactly your customer will be, exactly what product you'll solve, what the product will look like, between the initial idea and something you can actually take to market.

So it was really weeks of time spent in that room and speaking to customers before we really got to the first version of Human Loop. But the first version of Human Loop looked quite different to the product today. But the first MVP that was good enough that we felt we could put it in front of customers, I think took us about a month to build.

The way the first version of the product worked was at a time that people still had to annotate data to train a natural language models. It looked like a UI for annotation and in 2022, we realized that as the large language models improved, you wouldn't even need to have almost any annotated data anymore, and the paradigm was going to shift away from hand labeling data and fine tuning towards prompt engineering and maybe also some fine tuning.

Raza: There was a certain transition point when we built something ourselves, just as an internal hackathon, and realized just how capable these models have genuinely become, and speaking to a handful of customers just to understand what their needs were. And what we heard was the same consistent pain point. How do I know how this system is behaving when it is non-deterministic? How do I go and improve it, because I genuinely don't know how to evaluate it beyond eyeballing examples.

We had this idea that there was an opportunity to help the people who were trying to build applications with this. But we weren't sure how big the market was or how pressing the need was. And so we gave ourselves this sales experiment. We had an idea for what the product would look like. We sort of had a mock-up. If we can get 10 paying customers in 2 weeks, then we know that there's a really strong pull in this direction and the timing's right.

And so we kicked off that sales experiment and after 2 days we had 10 paying customers. So it didn't take 2 weeks. And we had never felt pull like that before. Like there was pull from our existing product, but not the kind of pull where you have the first conversation with someone and they basically like take my money. And so the fact that it was so strong, the market pull, and we were so confident about the overall technology direction, at that point made the decision to pivot reasonably straightforward. And it just felt like this was obviously a better trajectory to be on.

Raza: And there's really two parts to the product that we're solving for them. So one is everything related to prompt engineering, versioning and management, and the other is evaluation. The product team uses Human Loop as their development environment for developing their prompts. So they're in that UI, they're collaborating together on that, iterating on things, and their workflow before Human Loop is usually split between multiple tools. Often it's the OpenAI playground and Excel sheets and other things. There's no collaboration, there's no versioning, there's no history.

And I think, I guess if you say what's the aha moment for the product team is the realization that they as product leaders can drive the process of the development of these AI applications without being as dependent on engineers as gatekeepers. And then I think for the engineering teams it's bringing some of the rigor of normal software development to developing LLM features, right? These larger companies have to have confidence that the models are going to perform in the way that they expect in production. They're not going to say something embarrassing. They're not going to create any liability. They're going to do what they're supposed to do and that's hard to do with LLMs because it's subjective and because it's stochastic.

And so you know in traditional software engineering we're used to having unit tests and integration tests and there's a whole process around it. There's good versioning. We give those teams something equivalent for LLMs and the ability to measure performance.

Raza: So companies like Duolingo who are building LLM applications, Duolingo has so many use cases, right? They have this Duolingo Max product which is an AI chatbot that people can practice speaking with, but also they have to produce a ton of content internally for their applications that could potentially be done with AI assistance. So those are the opportunities and applications. But when they come to build these things, there's a lot of prompt engineering involved. The people who are best placed to do that prompt engineering are the domain experts, people like the linguists.

So you need some way to allow the linguists to collaborate with the engineers and to develop the prompts and then measure performance, and you have to have confidence that it's going to work the way that you expect when you deploy it.

Raza: I fundamentally think that AI startups aren't a different breed of company from normal companies, right, they're still building a product that should solve a problem for end users, but fundamentally, your end user doesn't care how you're delivering the product to them. They just wanna know, are you solving an important problem for me.

I think one of the mistakes that most startups make is they hire too many people too early and they hire people before product market fit. And if you have a team of 20 people and you don't have product market fit, it's really hard. It's really hard to experiment and iterate and learn from customers and change things, it's actually much harder to find PMF with a team of 20 people than it is to find PMF with a team of 3 people. You want to stay a small team until you're confident.

I guess another one is that a lot of startups right now are very excited about agents, AI systems that can take actions in the world and act on behalf of the customer, and I think that is certainly the future. That's definitely the direction of travel and very soon we will be there. But today it's still hard to make those reliable. And so a lot of people are trying to build something that maybe is slightly overambitious for the current moment, but correct for the near future.

Raza: So I don't think they're doing it wrong, but it's not surprising to me that it's hard to get it to work just now. But I think they're correct to be building for the capabilities of the models a little bit further on from where we are now, because they are going to improve very rapidly, and it's better to be building for that slight future point than be limited by the capabilities today.

If you're not smart about expecting where the future will be, you will be not beneficiary of these trends, but a victim of them. You don't want to be building something where if a smarter, better model comes out next week, your company is screwed. You wanna make sure that you're riding that wave of improvements.

The question was, what, how does it feel surfing on this giant wave of AI, right? Things are moving incredibly quickly. It's exciting, it's tough, it's challenging, but why would you want to do anything else? If you'll go out there surfing on a little small wave and you're seeing this big AI wave over there, you're gonna be a bit jealous.