Flying planes in Microsoft Flight Simulator with a JavaScript autopilot (2023)

For anyone interested, Asobo studio is currently hiring an Autopilot Programmer: https://jobs.eu.lever.co/asobostudio/1a940a26-31c4-4720-8271...

Honestly would never see something like this writeup from MSFS add-on developers. They are a protective bunch who share nothing with each other. This is in contrast to X-Plane add-on developers, who are friendly and collaborative.

I guess it also has to do with whether or not someone will go "oh cool, cool, okay I copied everything you did and quickly released a shittier version that made me a few bucks but turned the flightsim community off of the plane you're still working on" =(

I think a lot of it has to do with how MS has dealt with add-on developers in the past. After FSX was abandoned by MS, if you wanted new functionality in the sim you had to build it yourself. So add-on developers would all have their own propriety tech for things like Avionics, nav and the flight-model.

Today MSFS is constantly updated with new functionality, so you can just focus on your project, but old attitudes remain.

Also note that this isn't a "let's learn to implement this thing that I already know works", where I already worked out all the problems and am just giving you a write-up that lets you skip to the final result...

We're going to get things wrong, planes are going to crash in myriad ways, and we'll discover how and why things go wrong, to hopefully figure out how we can navigate those problems. And we're going learn from our mistakes.

Including learning to throw away all the work we did because that's the right thing to do =)

Nice! X-plane also has a great API to do these things with. It was created by NASA for research: https://github.com/nasa/XPlaneConnect

There's even a pretty great autopilot for x-plane post that makes it to HN every now and then!

https://austinsnerdythings.com/2021/10/15/creating-an-autopi...

Austin from the referenced post here - always love to see the shoutout! What you did is pretty much what I wanted to do when I set out. But definitely not in JS haha, not a huge fan of it

Anyways, I chose to go with PID control for my control systems, which worked really well right "out of the box" with some somewhat generic values for P, I, and D.

I read/skimmed all the way through your post. It looks like your control system is more or less a P & D controller? Any reason you didn't just go with PID loops? Is this part of needing things to be stateless to work for JS (I am no JS expert, just saw a few mentions of that in the conclusion)?

It's basically P-D-and-a-little-second-derivative, but the straight PD controller code just doesn't like the whole "you only get to run every half second" timing aspect. Things take far too long to ramp up, and the amount of overshoot you get is just ridiculous. There's no way (at least, not that I've been able to mitigate in anything even close to acceptably) to avoid super obvious oscillation, even if it eventually converges, at the low rate we're running.

Someone could, if they wanted (as one of the "what's left" options =D) replace all of this with PD code but they'll probably need to run at a much higher data rate for things to actually behave, and optimize the SimConnect calls to get the dataframes necessary without tying up a thread.

I haven't grok'ed things properly yet, its a hefty read for my Monday morning, but the thing that leaps into my mind is to ask you why Javascript, of all things, and not something like Lua?

Isn't Javascript just too finicky for something like this? Are other languages/approaches to integration with the execution environment possible/feasible?

(I apologize for my newbie question, hope you don't mind..)

What’s your definition of finicky? JavaScript is something like 3-10x faster than Lua for most benchmarks (1). If you’re referring to the event loop, it shouldn’t be all that relevant for singleminded code like this that is only processing a single event every half second. If you’re referring to mathematical oddities, it is also not relevant as JavaScripts love of floats is well suited here.

1- https://programming-language-benchmarks.vercel.app/amp/lua-v...

Thanks for the enlightenment, I can see I have to work on my understanding of JS a bit more .. I did not know that it surpassed Lua in terms of performance. I guess Mike Pall is sorely missed.

> why Javascript

Because no one's paying me, so my choice in language is about how easy I can do things like async code, hot reloading, and fast enough performance to not feel like the language itself is the bottleneck, and that makes JS a no-brainer (especially as Modern JS is quite fast, too!)

> Isn't Javascript just too finicky for something like this?

Maybe you're thinking of JS as a browser language, in which case: no, it's been a general purpose programming language with a standard library for over a decade now in the form of Node.js, which has been keeping in lockstep with the development of the language itself (which has changed a lot over the last decade, so if you've not looked at modern JS circa 2022~2024, it's rather different now =)

Thanks for taking the time to reply, I had not really thought my question through .. I'm going to set aside some time to experiment with your project in my own MS Flight instance, and see how far I can take my understanding of things without getting on your radar. ;) I appreciate the patience it took to explain my dumb question, in hindsight it of course all makes sense.

I see, yes controlling at 2 Hz will not be easy no matter how you slice it. What's preventing you from doing more? At one point I experimented with mine. I think below 5 Hz things got choppy. Greater than 10 didn't do a whole lot.

For reference, Arduplane (RC plane OS/autopilot/etc) uses 50 Hz for defaults - https://ardupilot.org/plane/docs/parameters.html#sched-loop-..., and quite a bit higher for copter (at least 400 Hz).

I'd be interested to know how your autopilot reacts with just the P & I portions. Everything I've read/learned about PID loops say to incorporate each term sequentially, meaning do P first, then introduce I to correct long term error, and finally introduce D to prevent overshoot. Not many use only P & D.

> What's preventing you from doing more

MSFS itself, mostly - it's pretty much always CPU bottlenecked already, and you can do fast polling using C++ (by basically allocating a piece of shared memory that MSFS keeps dumping updates into), but when you're using an FFI layer in Python, JS, etc. you end up transferring "fat" objects and running real time without bogging down an average specced machine gets tricky. The game itself will generally run well below 60fps, so realistically the fastest you're going to get updates is 20~30Hz (which should be enough) but then you might also have your AP code competing with MSFS itself if they decide to run on the same core.

It would be cool to add a completely new chapter going "so we got all of this working pretty well: time to delete all our code and now redo it using control loops on an unused core" but that's one of those "if I ever have time again" things =P

> Everything I've read/learned about PID loops say to incorporate each term sequentially

Aye, but the problem is we don't have the luxury of tweaking the controller ourselves beforehand: the goal is an autopilot "that works for all planes", so the challenge would be to write a self-tuning PID solution that can do the job without any prior knowledge (because we're not profiling every plane in MSFS, that'd be a full time job). If we were writing an autopilot for our favourite, single plane, then tweaking the controller would be relatively easy, but the solution has to work for everything from a Top Rudder Solo 103 to a DC-3, which complicates things a little.

I'm not in love with the current implementation in erms of having quite a bit of "emergency detection" that changes control parameters if the current values end up steering us into bad control territory, but at the rate we're getting updates, it's at least serviceable.

The best thing about Hacker News is that it provides all the knowledge you need to make your own suburban weapons system.

Last month it was radar [1], this month navigation!

Can you extend the tutorial to cover positioning from the elevation data? :-)

[1] https://news.ycombinator.com/item?id=39930195

Yes, an overview of INS, terrain matching, and vSLAM algorithms would be interesting. You know, just in case you're flying around somewhere GPS is being jammed right? On a serious note, I hope the Ukrainians who designed and built long range drones that can accurately hit the distillation columns of Russian oil refineries after flying hundreds km through contested air space get the chance to write some books about their accomplishments one day. It's quite the achievement isn't it?

I’m sure it’s quite a challenge, but I don’t think they are autonomously navigating directly to their targets. I’ve read they are using autonomous flight until a hand off to an operator behind enemy lines.

I think I'll leave that as an exercise for the reader =P

Wow, that is a lengthy tutorial.

I only skimmed over it now, so maybe I missed it: How do all these developed methods compare to existing real autopilot systems? Are there even existing autopilot systems which work similarly? Did the author come up with these methods just by themselves, or were they adapted from existing systems? Would those developed methods make sense in a real autopilot system?

Also, it seems that this involved a lot of trial-and-error, and the solutions come with a number of heuristics and hyper parameters to tune. I wonder if you can do better and avoid that as much as possible.

First of all, you would need to have an actual metric which covers all the things you would care about. Obviously no fatal crash, but then also maybe somewhat smooth flight (you can measure that) and how close you want to get to each waypoint, while maybe taking not too much fuel. And given those, let it find the best possible path. I guess this is optimal control theory then. When you have an actual metric which you can measure, then you can quantitatively compare all the methods, also automatically optimize any parameters.

I haven't read the tutorial (yet), but I know a bit about autopilots.

Autopilots depend a lot on which kind of plane it is in, but for the most part, autopilots aren't really "auto pilots". They consist of various fairly basic primitives rather than more complex decision making.

As a rule of thumb, the complexity of the autopilot is proportional to the size of the plane.

Basic autopilots can maintain headings, maintain (barometric) altitudes (i.e. above sea level) and similar very basic operations. For these autopilots it is the pilot's job to ensure the plane is operating within the envelope and does not depart from controlled flight and the throttle is controlled by the pilot.

As an example, the autopilot in an F-16 fighter jet has four things it can do: maintain attitude, maintain altitude, fly toward nav point and fly a compass heading. That's it.

Autopilots in airliner jets are more complex and can handle a set of nav point, ascent/descent, arrival and, depending on the airfield, approach and landing. They also feature autothrottles. Their implementation is still fairly simple, since it's a matter of matching throttle, attitude and heading with the position and altitude of each nav point. The autopilot doesn't have to do any pathfinding or more complex decision making, in order to keep it simple, predictable and bug free.

How does autopilots using barometric pressure maintain altitude, when these can vary wildly in non-calm situations? I would expect at least gps added to the mix (although with russia blocking skies randomly that may not be the smartest option these days). Or somehow keeping inclination (dont know the proper name of instrument here).

It uses the pressure altitude relative to the sea level pressure you configure in the plane (the altimeter in inHg / QNH). Above transition altitude (in the US usually 18000) you switch to standard pressure (29.92 inHg), so for those higher flight levels we ignore pressure differences. But for altitudes below this, yes you constantly have to change the altimeter in the plane or else your altitude / flight level chance / VNAV descent / climb will be incorrect. Of course you can always get the pressure by tuning into a ATIS / AWOS etc radio station for a nearby airport if you aren't getting the latest pressure from the ATC enroute.

GPS is usually not used for altitude.

Just to add for completeness, for ICAO/the rest of the world:

Transition altitude (ascent) and transition level (descent) is typically defined per FIR (flight information region), but can be airport-dependent. It is listed in the relevant airport charts. 7000 is fairly common, but it varies. Altimeter setting is measured in hPa/millibars (with the standard being 1013.25, which corresponds to 29.92 inHg).

To expand on verst's answer, the altimeter setting is defined based on your location, to roughly match meteorological conditions. When flying IFR or otherwise directed by ATC, you will be given altimeter settings. They will also be repeated during arrival and landing. In addition, they are given using an automated weather service called ATIS. ATIS broadcasts are given an incremental letter, that way you can verify you have the latest update. When checking in with a controller you will give the letter, and they will tell you if you're outdated.

The most important thing is that everyone in an area reference the same altitude/altimeter setting for traffic avoidance. Whether or not that is off from the actual altitude by a few hundred feet or not is secondary, since ground avoidance typically uses a lot bigger safety margins. Airliners and military planes have radar altimeters to measure height above ground when flying in IMC (low vis), while smaller planes typically fly VFR (based on visual rather than instruments).

> Airliners and military planes have radar altimeters to measure height above ground when flying in IMC (low vis),

But the RA are used primarily in the later phases of the approach. The barometric altimeter setting remains vitally important throughout most IFR ops in all categories of aircraft. (Radar altimeters are a key component of the ground proximity warning system GPWS in transport aircraft.)

> while smaller planes typically fly VFR (based on visual rather than instruments).

General aviation aircraft are quite frequently well-equipped with instrumentation for IFR ops. I’ve flown approaches down to published minima in single engine aircraft and have (legally) departed in low IMC conditions that grounded Part 121 flights. (Not necessarily best judgement…but here to tell the tale that I’ve become wiser in older age…)

> But the RA are used primarily in the later phases of the approach.

Indeed, the context of my sentence was using it for ground avoidance.

> General aviation aircraft are quite frequently well-equipped with instrumentation for IFR ops.

Yes, but most GA flights are VFR flights. Again, the context was ground avoidance and my point was that since they are usually used for VFR, more advanced means of ground avoidance are not generally required.

What about drones? They would need to have a complete solution to the problem, right?

Are there autopilot systems that do any sort of drone, bird, or other aerial object avoidance?

Then, aren't regional cultural ethics necessary to solve a trolley problem; when there are multiple simultaneous predictable accidents and you can only prevent one.

Does "autopilot" solve the trolley problem for the liable pilot?

There's collision avoidance systems, but those are really for "not flying into another plane". You're going so fast that by the time you can pick up that you're about to hit a drone or bird, you've destroyed it -and possibly yourself- already.

Military drones are a different matter: you will never find yourself in the same air space, because you will never be told they're there. They are ghosts, much like airplanes that run without a transponder "because it's legal". You don't see another aircraft unless you know exactly where to look, or you're right on top of them.

Flying things could sqwak with the inverse of the Doppler transform when subsonic

https://news.ycombinator.com/item?id=37886514 :

> FAA UAS RemoteID: https://www.faa.gov/uas/getting_started/remote_id

Collision avoidance in transportation: https://en.wikipedia.org/wiki/Collision_avoidance_in_transpo...

> I wonder if you can do better and avoid that as much as possible.

You can! However, that's basically another 100 pages of "and now let's throw all this code away again, because it's time to switch to self-tuning PD controllers with a drastically decreased polling interval".

Really an autopilot is a combination of as "super dumb, but effective" control processes (the dumber the better, because less code equal less bug equal more alive) that are ideal for PD control, but that does require running the control loop as close to "real time" as you can, whereas we started with "let's not bog down the system, and run twice a second" (because MSFS is programmed in a, let's call it "special" way, and it's basically always CPU bottlenecked)

So... exercise for the reader right now =D

Also cool, GTA V autopilot using OpenCV, Xbox input, etc. https://github.com/davuxcom/GTA-Pilot

This is an awesome read. Really enjoyed the level of detail and the various tries before things work.

A bit clickbaity of a title, but the material was far more interesting and in-depth than I expected, although there was no mention of "hunting" which can often occur in control systems like this.

What's clickbaity about it? The title is literally what we're doing: we're flying planes in MSFS using an autopilot that we're (progressively) writing ourselves, using JS.

The title was changed --- it just said "Flying Planes with JavaScript" or similar when I first saw it.

That's still the title of the actual post. I like it though.

Ahh, gotcha.

A question: I skimmed through MSFS and Xplane API docs but could not find a way to fetch full camera image. Does anyone know of a flight sim (except MS AirSim) which would allow that? Please?

Given how expensive that would be (it would be a staggering amount of data!), I doubt anything would allow for that. However, what you might be able to do is use the values that you read from one MSFS/XPlane session and then use the APIs available to replicate the same flight and flight properties in another, separate instance of MSFS/XPlane (on another computer, for example), and then capture its HDMI/DP-out

maybe the OSS Flightgear[1] flight sim might be a good place to start [2]

[1]https://www.flightgear.org/ [2] https://wiki.flightgear.org/Portal:Developer

Awesome. It would be a nice challenge to make a helicopter autopilot, because MSFS doesn't include one like it does for fixed wing.

I feel like one of the well respected add-on folks made a helicopter with autopilot, but yeah, it would certainly be an interesting exercise!

hehe.

"wasn't that the guy where I got my baguette recipe from?"

goes searching

"it WAS! beautiful. Then this guide here surely will also be fun"

Haha, that's fantastic. Trying to see if there is any way to tie baguettes, Bezier curves, and autopilots into a single thing, but until Microsoft releases Bread Simulator 2025 I might be out of luck there =D

We need to integrate this with Twitch so viewers can be my copilot… with the expectation of amusingly disastrous results!

That'd be hilarious =P

Something in the page kept reloading it when scrolled down trying to read on mobile firefox

Yeah, you really don't want to read this on mobile. It's 200 print pages, I have no idea how long you'd be scrolling.

(I need a better nav menu solution for mobile, at the very least)

I imagine you put a lot of time into this. That’s why I pointed out about the critical bug in this page - it sporadically reloads the page completely preventing it’s use on mobile. 50% of users are mobile users in the general case. Just my 2 cents

Okay that's.... there isn't anything on the page itself that would even do that so that should definitely not be happening.

Could I ask you to file an issue about that over on https://github.com/Pomax/are-we-flying/issues ?

If you are running MSFS on mobile... Seriously?

> To allay any concerns: this is not about running JavaScript software to control an actual aircraft. That would kill people.

SpaceX: Hold my beer (https://os-system.com/blog/javascript-in-space-spacex-devs-h...)

Mental note: hold off on considering taking SpaceX flights O_O!

To be fair all critical systems can be controlled by hardware buttons below the displays so they don't need a 100% guarantee, worst case rebooting would be fine

Yes, this .. and anyway, isn't the astronauts interaction during critical flight phases completely superfluous to the mission? The screens are there to tell the occupants how close they are to death, it seems, and little else.

Heck, for a long time - decades, even - the principle 'interface' to the rocket systems being used to regularly launch humans to space was, literally, a stick.

Javascript seems like a huge upgrade from a stick. ;)

It ain’t flying civilians so who cares.

Just dropping by to say: Hi Pomax! Long time no see, hope you're doing well!

_Fantastic_ tutorial!

Glad to hear you liked it! Buying that HOTAS from you "to see if that makes MSFS fun or whether I should just get it refunded" may have sent me down a rabbit hole that was deeper than I could have ever expected =D

Haha! that's so great, I'm glad the hardware found a good home :)

great sharing

> (barely an inconvenience)

You need to get aaaall the way of my back about why flying planes with JS sounds like a bad idea :-)

Plus it's in the script, so...