Hacker News Reader

Comment by engi_nerd | Hacker News Reader

I'm always amused when I see the criterion for a "very dense" time series as data being collected more than once per second. In my business (telemetry), we often record parameters thousands of times per second, depending on what we are trying to measure.

View on HN

I've got a system here capable of collecting data close to a hundred thousand times per second from tens of sources, with perceptual real time (< 10ms) processing and aggregate monitoring.

It's called multitrack audio recording, and commonly runs on your run of the mill laptop. HD video would probably be a few orders of magnitude more data than that, with even more processing.

Computers are really, really good at these kinds of things if only the software is efficient enough.

Your last sentence is well taken, but the rest leaves me a bit puzzled about what you meant.

Your run of the mill laptop is a wonderful machine capable of amazing things. But that laptop is not rated to withstand extreme vibration and shock and huge temperature ranges. Nor does it have the external interfaces to deal with large numbers of custom, high speed digital data buses, nor does it contain the signal conditioning hardware to deal with a huge array of transducers working with differing physical phenomena, each with its own unique power and signal processing requirements. The system I support has to deal with all of these things, plus more constraints than are relevant to the conversation here.

Much (but not all!) of this doesn't even require a general purpose computer. It can be done with state machines implemented in FPGAs.

Sometimes you need something different than a run of the mill laptop.

I apologize if it came across as a snarky comment or one-upmanship, I merely intended to support your original comment.

No problem.

Like I said, you do have a very good point. I am often frustrated by the realization that my laptop is much more powerful than this specialized data collection equipment. But we have a very large fixed base of equipment, with no plans to upgrade anytime soon.

Do you use a database for your data? We store our flight test data as individual DATAQ files with a naming scheme. This is manageable for our very simple civil aircraft so far.

Ah, a fellow flight test person. I don't handle the warehousing of data, so my knowledge is limited. I know a database is involved and used but I could not say what database or provide any specifics. Sorry.

EDIT: I would like to know how other flight test people visualize their test data. Seems like you can only make so many envelope charts and strip charts (they work and are effective, but man, sometimes I'd like something different)

All we use are strip charts. It's all I saw at Boeing as well (through some proprietary charting software IIRC). What I'd like is some "rich" time series data format that stores a history of modifications (removing noise, filtering etc.), and an overall better interface for scrolling through the data (Windaq is pretty barebones).

Wow, that is pretty rough. That's so rough that I would consider figuring out some way to export your data from that software and then visualize it with python/matplotlib or similar.

I've been working on a Python module to read the files directly: https://github.com/choj/cldumpread/tree/master/WindaqReader/.... Scipy/numpy/matplotlib totally opens up what we can do with the data. The hard part is convincing a mechanical engineering manager who's extremely averse to new tools.

I've been there. Once I had to jump through huge hoops just to be allowed to install Python on a machine not connected to any networks.

I remember talking to someone that had an in-house flight test data warehouse with these kind of features, I think it might have been Cessna - give their FT dept a ring :)

I use kst for plotting, transforming, visualizing, zooming, scrolling etc. It handles real time data nicely but I'm only doing 10 hz from a Plc using modbus protocol.

The collection time is only a single dimension, to see the scope you also need to know how many different pieces of data are being collected and for how many discrete groupings. Compare 100 inputs for a single device 1000 times a second to 100 inputs times 1000 devices once a second.

Fair enough. In the telemetry world the second case you mention is rarely seen. I'm sure someone, somewhere is doing something like that, so I won't say never. In flight testing you have anywhere from a few measurements to thousands of measurements gathered from physical transducers, and then digital data bus information that can vary dramatically from platform to platform, from just a few kilobytes/sec to gigabytes/sec.

There's many, many cases where you have lots of discrete inputs but a lower time threshold. In/out bytes per port on switches in a datacenter. CPU/memory/disk stats on virts you spin up for on-demand load, or just your normal datacenter deployment. Employees/vehicle locations in the field. Often you don't even need per-second time granularity, but that can be made up for with a larger number of discrete inputs (how many trucks does UPS have in service?)

Indeed, those cases are normal in other industries, but not in mine.

Not that funny anymore if you try to record 1MM time series at once. :)

I'm not anywhere in that realm because I don't have a data system that could support such a large amount of data. But then, I also have a physical system that does not need observations of 1 million distinct things (at least, not yet).

Can you be more specific? Is it car engine telemetry?

Aircraft telemetry.

You really need to know the position of an aircraft to a thousandth of a second?

I said nothing about position. But a lot of things measured have to do with how the structure of the aircraft responds to turbulence, rough air, and aeroacoustic vibration (aka flutter). So there might be modes where structural components have harmonics that are pretty high (several hundred to over 1000 Hz). Therefore you must use a transducer that has a frequency response that can cover that range, and sample the output of the transducer at least twice that rate (at an absolute theoretical minimum, but rule of thumb is 5x oversampling).

The highest sample rates I remember seeing was 20kHz, for the pressure sensors used in turbofan inlet distortion testing.

That isn't out of line with the ways I have seen that sort of thing measured.

Possibly he might measure some other thing, but still after your question I started to wonder how much a passenger plane moves per thousandth of a second.

So: some quick googling suggests that "economical cruising speed" of an Airbus A320 is 840km/h [1]. After quick back-of-envelope calculations, this gives ~230m/s, so 0.2m per 0.001s. Given some possible uncertainty of a single measurement, I'd imagine that's not unreasonable level of precision when e.g. your Airbus is landing on an airport.

[1]: http://www.airliners.net/aircraft-data/stats.main?id=23

I guess I should mention that GPS position is a perfectly normal thing to include in a telemetry stream.

+ ins position of course; gps position is not reliable enough (less gbas) to land a big plane with.

Reply on news.ycombinator.com