jashmota

These are really really interesting thoughts around the remote teleop interface. A few things you mentioned have been on my mind. Teleop UX is underdeveloped and its affects is underestimated, and it could turn out to be a huge thing for us and humanoid companies if autonomy is harder than it seems now. I don't believe construction equipment operation today is optimal. We just go with it as that's what we have. It's far from intuitive and easy for a newbie to start operating to give some context and requires hours of practice. There would be a bit of training to be done for remote teleop as well. Might as well make it easy to interface and improve on the existing experience, which hasn't really evolved for years and decades!

I'd like to have a chat with you if you're up for it: contact at useflywheel dot ai

I completely agree. If you have worked in the industry, I would like to talk to you and get your feedback on what I might be missing on my understanding in requirements in safety layers. Would be great if we can chat on email: contact at useflywheel dot ai

This is a great idea - I would dig into this. I disagree on AI not winning here. Support crew mostly work on eliminating blindspots for operator. Yes you can eliminate blindspots with cameras and a 7inch TFT screen, but those are small, don't work great in direct sunlight. The current solution not being 10x better is why they still fallback to having those crew members. I genuinely believe we will change that.

I totally agree, what I meant was if we only do CANbus retrofit, we would really limit ourselves in how compatible we are. So we have to focus on doing the right kind of retrofit really well. CANbus retrofit is easier and we've done that many times before. Especially because my cofounder Mahimana used to work in self-driving company and worked on this and single-handedly built out their current ML data and eval pipeline.

You've touched one of my favorite things to think about. I think about this a lot. There will always be people who enjoy fully analog cars/planes/watches over alternatives. I think that doesn't define the whole market. How many of new joinees are we seeing in this industry? Would you rather have a functioning back because you operate these remotely or go to site and sit in one of these and have a broken back by 50s? This is leaving out the fact that remote operation could save lots of lives. In our launch video we did put together some of the news clips on it: https://www.ycombinator.com/launches/O8i-flywheel-ai-waymo-f...

I agree - the dumping and digging itself (where you move boom, arm, bucket much more than swing/tracks) should be manual. But swinging to the truck and back to pit (pure swinging motion to center around these areas of interest) do not have to be manual. I agree with your and other comments that the transition has to be smooth and that's something we're working on.

1. We're OEM/machine-agnostic 2. Full compatibility with any existing model and not just specific recent models

I think swinging (which is about 40% of dig and dump workflow by time spent) should not be manual. That's lowest levels of autonomy which requires roughly centering to the pit/truck which we have already achieved. Hence operator only has to look in front!

Those workflow numbers come from multiple observations at different sites, one of the examples is this: https://www.youtube.com/watch?v=orEsvu1CS64

I wish to talk to you because it's rare sight someone has a Takeuchi - is there a way to connect? My email is contact at useflywheel dot ai

I'd like to add that this was done in partnership with Frodobots.ai (https://frodobots.ai) and BitRobot.ai (https://bitrobot.ai). They are working on a great mission statement and were amazing partners!

I definitely agree with you - recreating the scene in teleop is challenging. In excavators, however, it does make it better. An excavator has huge blindspots on the right (due to arm), to the back and sometime near the bucket. Hence the workers who are hired to stand around (banksmen, spotters, signalmen) and signal to the operator.

It's like driving a Ford F150 without backup camera. You'd add the backup camera upfront, and not display the back view at the back window.

It's definitely challenging and we're far from something that's perfect. We're iterating towards something that's better everyday.

I can relate to how you felt. Even though I developed this myself, when I saw it running remotely with no one sitting inside for the first time, it felt magical and weird. This is going to save lives and look magical and that's too much fun to put off and do something else!

The way I think about this - we should not have multi screens. Human field of vision is 60 degrees for central and about 120 degrees binocular. The bucket of the excavator is way narrower than this which means actual task doesn't require wide vision.

So if we are able to have really good autonomous safety layers to ensure safe movements, and dynamically resize remote teleop windows, you'd make the operator more efficient. So while we stream 360 degree view, we get creative in how we show it.

That's on the vision side. We also stream engine audio, and do haptic feedback.

Takeuchi are interesting! Rare ones to have blades even on bigger sizes - is that why you got one?

I would love to explore that - could you help make this happen (I'm guessing you know him)?

Mining has had autonomous haulage (and to some extent excavation) for some years. They have been very tightly coupled to the individual machines, models and the quarry site.

Construction excavation workflows tend to be different and more varied (hence you would see more variations in the machines/attachments available for this industry). And so far there's no solution which just works or is plug and play. This is an attempt to solve that.

And we believe this is great time to do so due to market demands too. No way Gen Z is going to do these jobs when you can make money being delivery partner/influencer, etc. The high-certified, high-skilled workforce is going to shockingly shrink in some years.

Haha yes - luckily we don't get this outside SF. We call it flywheel because our mission is to solve data flywheel problem in robotics.

related, this cool photo: https://x.com/jash_mota/status/1969635994220839095

I might be less experienced than you - I've operated upto 38 tons on maximum 15 degrees incline. I wasn't moving tracks a lot when I did that. Would like to hear what scenarios you'be been in and how would you describe your experience? Maybe I could try those out to learn more!

Interestingly, we had some people try out VR teleop: https://x.com/Scobleizer/status/1970245161306464667

https://x.com/jash_mota/status/1969091992140304703

I think force feedback is key for small excavators, but not really true for 25+ tons excavators. Hence how easy it is for operators to accidentally kill someone with it.

That's indeed what we're trying to test to the extreme - to see how far we could go with just vision. We haven't done heavy excavation workflows yet, but we have some early success with some excavation workflows with just vision input and joystick action output (even without joint angle feedback!). We're betting on having really huge data with compact observation input and experiment to see if it holds water. If not we can always dial it down and add more sensors/feedback.

Thanks! Yes we're testing both CAN spoofing and retrofits. CAN spoofing really limits compatibility to a few machines/brands. Would love to get in touch with you: my email is contact ai useflywheel dot ai

We're indeed streaming the audio and have haptic feedback. My hypothesis is the seat vibration isn't as helpful. It's sub optimal and operators would be far more productive without it. We would do a paper when we have enough data on this. We're also putting more cameras but streaming a lot of them at once is tricky.

You're right! This is exactly why we like to do mechanical actuation - we are able to achieve bilateral telepresence, which essentially gives the torque (haptic) feedback over the internet! So on small excavators, you can absolutely feel the resistance. We also stream the engine audio, which tells you how hard the hydraulic pump is working. Operators like our system for these reasons :)

I'd like to get a chance to talk to you and your Dad to get feedback. How do I reach you? My email is contact at useflywheel dot ai

Thanks! Let's chat: contact @ useflywheel dot ai

It's far cheaper than millions! We're making something equipment owners want :) Let's chat? My email is contact at useflywheel dot ai

You're spot on! The experienced operators are quite skilled at moving these machines. The number of operators who could do this work are decreasing and it's also pretty dangerous which explains the reduction of new people joining in, especially with better ways to make money today. We're getting started with remote teleop and moving towards full autonomy!

We have some autonomous safety layers (comprised of object/human detection like you mentioned and others) even for remote teleop. But we haven't deployed autonomy on real sites yet. We still have some work to do before we can reliably deploy/test it out on the actual sites!

It is possible. One good thing about our retrofit is it works on any hydraulic equipment, and not just excavators, although we are focussed on excavators for now. We are able to retrofit any hydraulic machinery because we actuate the joystics connected to the valves, which are quite standardized and usually come from handful of companies like Bosch. Will definitely 'dig' more into this though.

Thanks! We enjoyed a lot while iterating the hardware and looking at hours of excavator data (mostly done by Mahimana).

DDG email is a service which acts as a barrier between you and people you receive emails from. It removes all trackers from your emails. However, to send email from this @duck.com email address, you need to tweak the username a bit.

This tool takes in the relevant parameters and generates that email address for you. When you send the email from to the address, it goes to DDG server which send the email from DDG ID.

So your real address is not exposed.

Even I clicked on it hoping it to be a an engineering concept related video.

But interesting video nonetheless.

Interesting. Have you thought of adding a way to show off the wins? Like people do with wordle? That will help in growing fast