Here's how Bryen Param built a drone you can talk to — and why he almost didn't believe he'd won Best of Live Agents in Google's Gemini Live Agent Challenge.
Bryen Param has always wanted to build machines you could just talk to. His project, Drone Copilot, won Best of Live Agents at the Gemini Live Agents Challenge created by Google. Here's how he built it, what broke along the way, and why he almost didn't believe he'd won.
(Check out Bryen's Devpost profile to see his other projects.)
I'm an AI developer based in Paris. Right now I work in data and AI for an energy company. The idea started from just watching how good AI models were getting and wondering how we could get them to interact with the real world instead of just living in a chat window.
I kept thinking about Iron Man! Specifically how Tony Stark talks to Jarvis and Jarvis just handles things. I wanted to see if we could actually do something close to that. So I thought: what if we gave a model a drone?
There are real applications past the demo part too. Warehouse inventory, inspecting large infrastructure, even search and rescue. The core idea is the same in all of them: you talk to the drone, and it flies and figures things out itself. No remote control. Everything runs through Gemini.
It was the "live" part. Not sending a prompt and waiting for a response back, but actually talking to the model in real time, sending it video frames, and having it talk back to you. That was the Jarvis moment for me.
I'd done hackathons before, but always software only. This was the first time I tried actual hardware. New territory, and I was curious whether it would even work.
You talk to the drone, and it flies itself — there's no remote, no manual commands. You can ask it "what do you see?" and it'll describe the scene back to you. Say "inspect that plant on the table," and it autonomously searches for the target, approaches it, takes photos from different angles, and generates a report. Or you can give direct commands — move forward 40 centimeters, rotate left, rotate right.
Wind, mostly. The drone is small and light, with just a 720p camera and no other sensors. I was testing in my backyard, and any bit of wind would push it off course mid-flight and cause me to redo the entire test.
The camera caused a second problem too: it uses the video feed to hold its position, so in low light it would drift upward on its own. That meant I couldn't fly indoors either, just outdoor only, and only when the wind cooperated. A lot of testing was just waiting for the right conditions.
Yes! I wanted the drone to navigate behind an object on command. Say "take a picture from behind the plant," and have it get there on its own, adjusting for different object sizes.
The problem was depth. The models don't have strong spatial awareness, so they kept misjudging how far to move to get behind something. I ended up hardcoding fixed values instead (move 20cm right, 40cm forward) which worked, but wasn't adaptable the way I wanted.
I actually brought this up with some Google engineers at Google Next. Most suggested training a model on custom data. One had a different idea: take a 360° scan of the space first and let Gemini build a rough 3D understanding of it before attempting the approach. Haven't tried that yet, but it's next on my list.
I mostly tested with a shoebox, since it had handwritten notes taped to the sides. After the drone approached and photographed it, Gemini correctly generated a report which identified the box, noted its condition, and picked up the handwritten notes. That part worked better than I expected.
Not everything did, though. I tried pointing it at a bicycle and asking it to focus on just the seat, and it couldn't isolate that. The object was too big relative to what it was trying to pick out. And reading the handwritten notes wasn't always reliable either; low light plus a 720p camera plus any motion during the frame capture meant blurry shots the model couldn't parse.
Honestly, my first reaction was that the email might be a scam!
Honestly, my first reaction was that the email might be a scam! I checked LinkedIn to confirm the people who'd emailed me were real before I let myself believe it. So many people entered, with genuinely strong projects, that it didn't feel real at first.
Getting to present at Google Next made it click. I met another builder there who'd also won in the challenge, and just being around people who'd gone through the same thing made the whole thing feel real.
Better spatial awareness is the big one. Trying that 3D-scan approach, or other models entirely. I also want to close the gap between a voice command and the drone actually executing it; right now the demo hides that lag through video editing, and I want it gone for real, maybe with faster local models. And better hardware — this drone was genuinely difficult to build with, and something bigger with more sensors and a better camera would open a lot up.
Longer term, I want it fully autonomous in a broader sense. Not just "inspect this object" but handling a task end to end. Infrastructure inspection is the application I keep coming back to, since drones are already used there but require trained pilots. Making that accessible to more people is the goal. Search and rescue came up a lot too, when I talked to people about the project, use cases like adding thermal sensors to detect people, for instance. A lot of people who saw it shared their own ideas for where it could go. It just needs to get a lot more reliable and secure before any of that is real.
You basically have a PhD-level collaborator sitting on your laptop.
Just build it. I didn't know going in if this was a good idea. I just wanted to see a drone fly itself, even if I didn't win anything. Flying a drone manually is genuinely scary; the idea of it flying itself was what pulled me in.
I also didn't know how to work with drone hardware before this. I think that matters less now with the AI tools available, you basically have a PhD-level collaborator sitting on your laptop. You can't build literally anything, but you can build a lot more than you'd expect. Be curious, and follow it.
Be curious, and follow it.
See Drone Copilot on Devpost, check out the code on GitHub, and connect with Bryen on LinkedIn
Explore more winning projects from the Gemini Live Agent Challenge
