Arduino-Based Home Weather Station on the Elastic Stack | Hacker News Reader

Arduino-Based Home Weather Station on the Elastic Stack

elastic.co

The BMP180 temperature sensor sports a +-2°C temperature accuracy.

That's too much for my sensibilities.

The DHT22's accuracy is +-0.5°.

Does anybody know of more accurate sensors (+-0.1°) suitable for usage wit the ESP8266?

Accuracy is not as important as precision though. For accuracy, you can usually just calibrate it out against a known reference (you usually need to do this anyway to get consistent results across multiple sensors). The only case when you can't is nonlinear accuracy, which is rare but I've seen it with the DHT22 actually.

Precision (stability of measurements) is what you really care about. That said, the BMP180 isn't so great at that either. The temperature sensor is I think a secondary function used to improve the pressure estimate. It doesn't give very stable temperature estimates in my experience.

If you want a direct replacement for the DHT22 (and I did, god I hate that module for a variety of reasons) the HTU21D-F is great. The BME280 is also a big improvement over the BMP180. If you just care about temperature, the DS18B20 is the way to go on performance and price, while if you really want high accuracy and precision then the MCP9808 is the way to go. I use the MCP9808 as my reference to calibrate against, with a mixture of the other three in various applications.

You seem to have experimented with some sensors. I was wondering what would be your top choices of sensors for someone learning hardware hacking with Arduino/Raspberry Pi(not only temperature measurements). Thank you very much.

Well, my favorites are:

* DS18B20 - One-wire temperature sensor that is cheap

* Generic PIR (motion) sensors - you can find simple motion sensor boards for cheap and they are pretty fun.

* Capacitive sensors - you can make a touch sensor/button out of anything conductive that only uses a single wire (not even ground, just one wire)

* Ultrasonic distance detectors

* Hall effect sensors - detect presence of a magnet, especially useful with stepper motors

* BME280 - temperature/barometric pressure/humidity, one of the best all around sensors for weather monitoring as it's all on one chip. Also supports both i2c and SPI. Note it's BME280, the BMP280 doesn't support humidity.

A good bet is to buy a kit that gives you a random bunch of sensors to play with. You can find them cheap.

Adafruit sells the MCP9808 (±0.25°C), not quite 0.1°C but closer… https://www.adafruit.com/product/1782

Why do people buy stuff from Adafruit when they can get them from eBay for a tenth of the price? Shipping speed?

Because Adafruit makes it easy to find and usually has some amount of tutorial/howto easily findable from the product pages. Not everyone knows that these parts can be sourced more cheaply; not everyone cares. Plus, there's some higher assurance that you're going to get what you pay for from Adafruit.

For that matter, why buy from Ebay when you can often get them cheaper from Aliexpress? (Or even cheaper from Taobao in a lot of cases. For me, Taobao is too hard to access as a native English speaker, so Aliexpress is where most of my cheap components come from.)

When you need one, fast, and want it to work, get it from Adafruit and you will be sure.

When you need 10, then its time to look at ebay.

Also, Limor is a force of nature. You know how to use half of that random chinese stuff you find on ebay because Adafruit or Sparkfun went there first and left libraries in their wake.

Because they write a lot of software and tutorials on multiple platforms for the devices they sell and people are willing to pay a little more to support them.

...how have I not noticed the "learn" link before? Thanks.

I buy stuff from Adafruit because they do good work. They are advocates for approachable and understandable electronics. They work hard at combining art and technology and education. They pick good components and they make software libraries and tutorials to match.

I buy their stuff because I think beginners should buy their stuff and I want to see more companies like them exist.

It's more about getting what you pay for rather than relying on your expertise to get something working. Plus Adafruit writes good software and tutorials to get you up and running.

When you are going in for a hobby project, you tend to go towards a more 'established' and 'complete' solution including software than having to rely solely on your expertise.

Pretty much this. I'm very new to microcontrollers and hobby electronics but I've been doing some projects with addressable LEDs and such over the past year. Adafruit is great for tutorials and forum help and they ship quickly in the US and have some level of protection against defective gear.

When I need to order reels and reels of LED strips? Sure, if I have the time I will hit up Aliexpress or something like that because I can get a few 5m spools for the price of half a spool from Ada.

But if I need a part that's $5 on eBay/Ali and $10 on Adafruit, I'm just as likely to spend the extra money for fast shipping and support even though the relative markup is high. It's kinda like how I'd still run out to Best Buy or Radio Shack on the rare occasion that I just wanted some small connector or part that day rather than waiting 2-5 days for Amazon to ship it to me for less.

Sometimes you can't source stuff from A/X, especially for more special purpose parts. I've never been able to find for example the MCP9808 on A/X for cheaper than the Adafruit price, though you still have to pay shipping (but you get it sooner in compensation). I also think the Trinket Pro is a reasonably priced Pro Mini clone, especially because a lot of the super cheap clones have awful voltage regulators on them. That doesn't matter in a lot of cases, but sometimes I need it.

I agree that a lot of stuff is pretty overpriced, but they are selling premium quality on a lot of their stuff, plus reliable supply chain and lots of support.

I sometimes buy from them because it guarantees a quality minimum. They have a good reputation to uphold and will make sure I'm satisfied in case of any issue. Buying electronics from ebay has a higher risk of receiving bad parts. Which means the project is set back and my time is lost. Id rather pay more and save time/headaches.

Because the eBay stuff is usually a copycat of Limor's work.

Do you tribute the artist or the guy selling the artist's bootleg CDs in the alley?

HTU21D comes to mind, has something like +-0.3°C, nice recent I2C device

edit: if you got time buy on Aliexpress/ebay, ada charges 4-5X for most stuff

I haven't see anything with 0.1°, but the BME280[0] has +-0.5° accuracy and also measures pressure.

I have a few DHT22s and find them to be a little bit temperamental (humidity drifts, sometimes won't return a reading) but I've been happier with with BME280 (though I've always used the breakout versions[1] as it looks like a bitch to hand solder).

It's an I2C parts and there's support for it in NodeMCU and Arduino so it's easy enough to get started with.

[0]https://ae-bst.resource.bosch.com/media/_tech/media/datashee...

[1]http://www.aliexpress.com/item/1PCS-Breakout-Temperature-Hum...

PT100 thermistors are the usual reference standard once you move away from the triple point of water.

It's been a few months since I last thought hard about ours (which we use in a different configuration), but 100 mK ought to be straightforward with a Protovoltaics RTD shield and a three or four-wire PT100 or PT1000.

Protovoltaics has a well-documented API and is quite responsive to detailed engineering questions.

Alternatively, one can calibrate a sensor against a trusted standard. Once that's complete, they tend to be fairly stable. LM35s and AD590s will have low-enough noise/good-enough stability for your spec once calibrated.

Before you embark on such a journey, be certain you actually want 100 mK absolute. It's possible to go through life happily without absolute calibrations in most cases.

Thanks for the information. Im not familiar with the intricacies of temperature sensors. But since every home weather station measures in 100mK, I've assumed that that's the precision I should get out of an ordinary sensor.

But you're right: I'm not entirey sure if I need this kind of precision.

There's an important difference between precision and accuracy. An LM35 will easily yield ~10 mK precision, but in absolute accuracy, they're far worse (the extent of the calibration error varies by model).

The Sensirion SHT35 [1] is good down to +/-0.2C. However, it's important to note that having such an accurate sensor is somewhat useless if there's not a radiation shield. A sunny day will change the temperature reading by several degrees C.

[1] https://www.sensirion.com/products/humidity-sensors/digital-...

Maybe one of these:

http://uk.farnell.com/ist-innovative-sensor-technology/tsic-...

Nice weekend project -- How often you are sending data to Elasticsearch and how does Kibana is polling the Elastic?

Another good combination would be Influxdb + chronograf

https://influxdata.com/time-series-platform/chronograf/

Or rather InfluxDB + Grafana.

Grafana is more mature and Chronograf is closed-source, I don't see any reason to go with it.

But yes if you want to display time series there are better options than Kibana, which has a focus on searchable documents.

Elastic.co is trying to get into that business. A time series datapoint is just a very small document, and they're churning out agents that collect and ship it.

I'm not entirely sold on the idea, as their examples for turning metrics into actionable insights are a lot more complex than they are for Grafana (or similar).

(Hi! I'm the original author!) I'm able to get about 200ms resolution at both ends without any trouble.

It seems reasonable to point out that there is no 'Arduino' involved here - it's an ESP8266-based board programmed using the Arduino IDE with the ESP-8266 board support package installed.

Its a pure Time Series data with sensor is recording approx every 1 sec. Which protocol you are using to send this data to server , as posting every 1 sec oever TCP for a sensor doesn't seems a good idea. For any IoT which protocol you guys prefer to push data to server

It all depends what you're doing. For a weather station you could do very well and have very low power at one reading per minute. HTTP is great for that.

Also your approximation is off. I'm under 200ms (closer to 110) per insert, end to end. That's before any optimizations.

Should you rely on es as your primary db?

Well my rule is basically - if search functionality is the most essential to your platform and the only other functionality you need otherwise is basic key-value stuff (look up items by user id, stuff like that) then yes, and handle performance issues with your preferred caching solution - redis in my case.

If you need something where money needs to be maintained then you need the security of relational database.

If search is only of secondary importance, use whatever database your prefer to work with and then index that.

But I also have the opinion that as time goes on search becomes a primary functionality of any platform.

Good question! For some toy weather data, Elasticsearch as the only data store made sense. If I were to add more sensors and features and make this a long term project, I'd likely try to wire up some automated backups. If it were some kind of production app that a second human relied on, then I would think hard about the data, the failure scenarios, budget, etc.

Does that answer your question at all?

It depends on what you're doing?

no, no and no

No, not for writing. It's never safe. It will eventually eat your data.

But for reading it's fine. So by all means, you can populate from a main data store that you never otherwise touch.

Then again, if you're running just a single instance on really small data... What could go wrong?

Edit: Again, small stuff, locally, it's probably fine. But ElasticSearch wasn't really built to be a primary data store.

Sources:

https://aphyr.com/posts/332-jepsen-crate-0-54-9-version-dive... (This has links to other good sources, right in the beginning).

https://news.ycombinator.com/item?id=11325316

https://news.ycombinator.com/item?id=11362069

yeah, I agree if your requirements is data consistency then not Elasticsearch, however I think from the Jepsen tests I don't think there has been any NoSQL db that performs really good on that? ( I seem to remember there was one that did bad on the first test but improved a lot with the second - SOLR, CouchDB?)

you're thinking of RethinkDB https://aphyr.com/posts/329-jepsen-rethinkdb-2-1-5

Thanks! I knew someone would have either a better memory than me or the willingness to actually go to http://jepsen.io/analyses.html and figure out which one it was! And now that I went I see both the ones I thought it might be haven't even ever been done!

I need to run a Jepsen test on my brain.

Reply on news.ycombinator.com