Show HN: Real-time server monitoring in your browser | Hacker News Reader

Show HN: Real-time server monitoring in your browser

scoutapp.github.io

346 pointsacl115 comments

They said they use it on their own production servers so of course I couldn't resist... http://scoutapp.com:5555/

Really nice looking monitoring, though. I think it's fun to see the stats scrolling by.

Ha yeah - we opened up a port in iptables for this so we don't have to ssh tunnel in to see stats. Obviously, if you're concerned about exposing numbers, you can view via an ssh tunnel instead of opening a port.

I see you are using SVG to render the charts. What are your thoughts on SVG vs Canvas for these types of apps?

Not these guys, but doing realtime viz of signals.

We've switched over to a minimalist canvas renderer--if you don't need interactivity or styling, and instead just "draw me as much as you can as fast as you can, damnit.", we hope it's the way to go.

I've been playing around (http://yield.io) with Flot, which uses canvas and rendering speed seems pretty good, but resizing gets a bit wonky when there are multiple canvas elements on a page.

Yeah, we started with Flotr2...too many graphs on a page (with thousands of datapoints per graph) brings Chrome to its knees, even with auto margins and whatnot turned off.

EDIT: Very clean site! I like your style. :)

@angersock That's a good point on the number of points. If I downsampled the yield history to monthly yields, that would probably help the performance.

that 2.6 kernel tho

Interesting that they are using kernel version 2.6.32-35. Is it common to run such an older kernel version on servers?

2.6.32 is one of the longterm maintenance releases; it's still supported and new security patches are backported. If you rent a freshly imaged RHEL/CentOS server today, that's the kernel you'll be getting. Pretty much all the software packages those distro's come with are older, longterm/stable releases, never bleeding edge.

For all my fellow CentOS/RHEL users here is a quick bash script for getting it running (assumes you don't have ruby 1.9 installed).

    # Install RVM and Ruby 1.9.3
    curl -sSL https://get.rvm.io | bash -s stable
    source /etc/profile.d/rvm.sh
    rvm install 1.9.3

    # Install JSON gem
    gem install json

    # Install Scout realtime
    gem install scout_realtime

    # Start Scout realtime
    scout_realtime

    # Punch a hole in iptables on port 5555 for Scout realtime
    iptables -A INPUT -p tcp --dport 5555 -j ACCEPT
    service iptables save
    service iptables restart

You don't need to install json on 1.9.3, it's already in the stdlib http://ruby-doc.org/stdlib-1.9.3/libdoc/json/rdoc/index.html

> curl -sSL https://get.rvm.io | bash -s stable

no.

Sorry to be the one posting a tangent (looks like a nice tool!), but please rewrite "for the modern man" to something more gender neutral.

I agree with this. I know "for the modern man" is an idiom, but I think it's better to be overly conscious about this stuff. Would you lose anything by saying "for the modern dev"?

Political correctness is such a depressing drain on the world. Leave it how it is. It's fine.

Reminds me of when they tried to rename Manholes to "Personnel Access Units" to avoid offending women.

If you're offended that something is named a 'manhole', you probably have too much time on your hands.

Let me guess, you're a man?

I'm assuming you're going to disregard their comment because of their gender? What does it matter what gender the parent comment poster is? They should have a voice too...

Actually, yes, I do tend to disregard claims that "everything is fine" from those who are not being aggrieved.

How would you respond if theknown99 turns out to be a woman?

Actually I'm not sure. However, I certainly don't want to "out" theknown99 as either gender.

I will add an addendum to my response as it is overly aggressive.

Well, since we're on the issue of political correctness, theknown99 could be transgender too :)

Wow, that was a hell of a sexist remark.

Ok, it's too late for me to edit, so please forgive my hastily aggressive response. What I meant by it is that it is easy to dismiss complaints about "one-sidedness" if things are tilted in your favour.

The word "manhole" is not a fair comparison. In that case they were dealing with an inanimate object. In this case, we're labeling any developer who would need this tool. Hence the inappropriate use of gender.

What if a website with nursing resources was subtitled "...for the caring woman"?

The landing page is on the gh-pages branch of the repo. I filed an issue for it here: https://github.com/scoutapp/scout_realtime/issues/6

Only reason I didn't file a PR myself was I wasn't sure what the best replacement would be (dev? sysadmin?), but this should be easy enough to fix if they're amenable.

EDIT: "Developer" as an alternative was merged very quickly. https://github.com/scoutapp/scout_realtime/pull/7

Sorry to be the one posting a tangent looks like a nice patch! But please rewrite "for the modern developer" to something more career neutral.

I'm a systems architect. I feel like I am not invited to use the tool.

Snark for any interested sentient entity.

Don't want to exclude luddites, transhumans, or extraterrestrials. I will settle for excluding moss and whatnot, though. Seriously, fuck moss.

::cries::

I can't tell if you're trolling or not...

The use of man in this context is androgynous. It's merely an abbreviated use of "mankind," which is an abbreviated use of "humankind." I'm sorry if you're upset by this wording, but it's not inappropriate.

No, you're right... it's not inappropriate. But really, something like "for the modern admin" is more appropriate and should be used. There is no need for the "modern man", so it should be left out.

The phrase of "modern man" was meant to contrast with old-school admins. For me, it brought up images of classic admins in server rooms either your straight-laced IBM types or your Berkeley Unix neckbeards. Take your pick, but they were both predominantly men.

Why not adminess? Or whatever female version would someone come up with after you'd use "admin".

Female suffixes like that can give the impression of the female version being secondary to the "normal" male version.

"for the modern man" is a very gendered idiomatic phrase unrelated to mankind.

https://www.google.com/search?q="for+the+modern+man"

Note how all of the results are all about men and manliness, which is fine for gendered products and clothing, but doesn't really make sense for a piece of server monitoring software.

i think whats interesting is that people who use sentences such as "for the modern man" don't think about man as male at all and don't intend to offend anybody.

It's only picked out by the ones who feel oppressed by gender issues (which are often males defending females - in fact, genetics also makes us behave that way, ironically.)

Of course they don't think about, and obviously they don't want to offend, and that is why some good people point this out, so that people think about it and pay attention. Sometimes biases are so entrenched that we don't feel them. They feel natural, and therefore neutral. But, if you want to make a change, than it's precisely those seemingly natural things that you need to change.

You can keep using "man" or not, but I think it's helpful to pay attention.

Frankly, i think some people just like to complain about gender issues when there isn't much going on.

In some languages (ex: french), everything defaults to male-centric. Nobody cares or feels offended by it, and females have exactly the same rights as males.

Feminism isn't just about not offending people or giving women the same "rights" as men. You can make the (true) claim that blacks in America have the same rights as whites. But does that mean that there's no more racism? And even without the judgmental word "racism", does that mean blacks have the same opportunities as whites? I don't think so.

Feminism is about making sure women have the same opportunities as men not only by virtue of the law, but "on the ground"; that society doesn't gently (or not so gently) steer them in directions where they end up with less power than men; that they're no longer objectified and that female politicians are not called by their first names.

I'm not saying language can fix all that, or that it even matters all that much. It certainly matters less in cultures where feminism has had greater success. But it is a good place to point out how, perhaps inadvertently, we keep falling into the same gender traps. If you start thinking about your choice of words, language becomes less natural, so you stop treating it, and the culture it articulates, as "nature", and start treating it as the malleable social construct that it actually is.

re: "for the modern man"

> The use of man in this context is androgynous.

No, its not, even if it was intended that way.

> It's merely an abbreviated use of "mankind," which is an abbreviated use of "humankind."

"Humankind", "Mankind" (and "man" when used in a sense that is semantically-equivalent to the other two) are mass nouns that do not take articles ("the" or "a"). In the case of "humankind" or "mankind", using an article is just plain incorrect (consider, "one giant leap for the mankind" vs. "one giant leap for mankind"), whereas for "man" the use of an article can distinguish between the sense of "an individual adult male human" and "humankind" (consider "the story of man" vs. "the story of a man".)

So, no, in "for the modern man", "man" doesn't work as a shortened form of "mankind".

You're thinking of "modern man", not "the modern man".

That's the first thing I thought. Please, please change it. Things are difficult enough for women in technology without things like that. I don't care what the rational is. It looks awful, and I'd be embarrassed to show that to women at work.

I honestly can't tell if you're trolling or not...lol...

It's a little too earnest - I think you may have overdone it...haha - too obvious.

Good job though.

Look, I'm all for encouraging more girls in STEM, and all that.

My EE class mostly guys, and I'm sure we lost some diversity of viewpoints because of that.

However, let's just ask nicely once - and then leave it at that.

I would hate this to turn into another silly bike shedding flamewars on HN, where all the Social Justice Warriors come out of the woodwork, for their weekly feel-good topup.

Too late.

Project author here. We don't want to exclude anyone - I updated it to "developers."

Quick question. How much CPU usage/time does the monitoring tool itself utilize ?

We've clocked the CPU usage of the scout_realtime daemon at 1% on an Intel Xeon 2.40GHz CPU. Memory usage is around 22 MB. If you turn off the metric collection (by clicking the pause button on the web page), CPU usage will effectively drop to 0%, and you'll still be able to visit the web page and re-enable metrics at any time.

I'm a sysadmin - can I still use your tool?

Of course not.

Do you think we want admins sullying our glorious code?

Have you thought of writing the metrics collection part (and eventually the whole thing) in Go? You would get the stand-alone distribution right away and would keep people from installing any extra dependencies.

EDIT: Fixed typo.

Already done!

https://github.com/gollector/gollector

Yeah - we thought about this, but decided to get started in Ruby since it's the fastest way for us to ship. Go is definitely interesting.

http://gfycat.com/JealousCloseAsiaticlesserfreshwaterclam

Just change it to "modern woman" and shut everyone up.

That's the thing now right? Where the english language has left us with lack of a non-awkward sounding gender neutral term we just the feminine version and it's ok. I know, I know, everyone is going to chime in with their version of a 'non-awkward sounding alternative. But the person who wrote this, wrote it, it didn't go to the committee of HN, and that person wasn't out to offend anyone, so ya know, let it go, let live, all that... No? I tried.

How about ...for the modern admin?

Can you guys add a if/else clause in sinatra and add a FreeBSD logo?[1]

I see tux in my FreeBSD server and feels weird.

[1] https://github.com/scoutapp/scout_realtime/blob/master/lib/s...

ps. My photo-editing skills suck bigtime otherwise I'd do it.

EDIT: Doesn't seem to work properly under FreeBSD-10. No data is displayed. Apparently (as expected) uses Linux ProcFS structure to get data. So FreeBSD for now is not supported, keep the icon for later :-)

I wonder if it can use the linproc compat stuff like htop does on FreeBSD.

Sorry - no FreeBSD support yet.

This is cool. It looks like the developers are reading the comments so I'll add a quick suggestion. Something that I found to be insanely helpful with my own product development was being able to track memory swapping.

I went to great lengths to tune my Java Virtual Machines so that they would work well in a minimum RAM environment. And being able to track swapping was critical for my decision making. Now I can run my product on a 512MB system with 1GB of swap space with no problem. Below is how I'm tracking swapping in real-time.

http://screenshots.gitsense.com/track-swapping.html

Since your solution is focused on capturing a period of time, you'll be able to provide a better view than I am.

With SSD becoming more common for cloud hosting, using swap space in lieu of getting more RAM will probably become more common. And before anybody points out that SSD is still significantly slower than RAM, I know. Depending on your product, using swap on SSD may be practical. I know using swap on amazon's infrastructure wasn't.

Part of the scout_realtime team here...swap is important. Displaying it the future is possible.

In fact, fire up the console on the project homepage and type "metrics.memory". We're capturing it, just not displaying it yet on the screen.

"We're capturing it,"

This is good to hear. Not sure what would be the best way to display that information though.

Nice, but it requires ruby :S

I been using this https://github.com/abimaelmartell/system_monitor, easy to install and dont eat too much ram :P

[2014-03-10 15:04:45] ERROR TypeError: nil can't be coerced into Fixnum server_metrics-1.2.0/lib/server_metrics/collectors/memory.rb:81:in `+'

Getting this error under OSX

Here as well, running with Ruby 2.0.0.

Sorry - mind sharing the output of "top -l1 -n0 -u"? Looks like an issue parsing out the memory stats on OSX.

Error:

  [2014-03-10 15:38:12] ERROR TypeError: nil can't be coerced into Fixnum
	/usr/local/Cellar/ruby/2.1.1/lib/ruby/gems/2.1.0/gems/server_metrics-1.2.0/lib/server_metrics/collectors/memory.rb:81:in `+'

Here's the output:

  [~] top -l1 -n0 -u
  Processes: 268 total, 3 running, 5 stuck, 260 sleeping, 2070 threads
  2014/03/10 15:36:26
  Load Avg: 1.92, 2.05, 2.08
  CPU usage: 4.46% user, 12.50% sys, 83.3% idle
  SharedLibs: 60M resident, 0B data, 4156K linkedit.
  MemRegions: 73352 total, 3529M resident, 83M private, 892M shared.
  PhysMem: 6726M used (1812M wired), 199M unused.
  VM: 570G vsize, 1312M framework vsize, 1570984(0) swapins, 1861805(0) swapouts.
  Networks: packets: 2441317/1583M in, 1428870/324M out.
  Disks: 3446592/122G read, 1578215/101G written.

Thanks - I can reproduce with that output. Opened an issue on github - we'll fix:

https://github.com/scoutapp/server_metrics/issues/9

Thanks again for reporting - we've released version 1.0.1 to fix the issue:

gem install scout_realtime

Note that OSX support is limited as there is no "/proc" support.

Processes: 180 total, 2 running, 7 stuck, 171 sleeping, 965 threads 2014/03/10 16:24:55 Load Avg: 1.28, 1.13, 1.06 CPU usage: 8.0% user, 20.0% sys, 72.0% idle SharedLibs: 12M resident, 12M data, 0B linkedit. MemRegions: 35120 total, 1588M resident, 121M private, 700M shared. PhysMem: 5020M used (853M wired), 2884M unused. VM: 401G vsize, 1065M framework vsize, 311(0) swapins, 1527(0) swapouts. Networks: packets: 14047540/18G in, 8183740/2177M out. Disks: 386988/26G read, 413161/25G written.

Nice. The play/pause should be one button that changes icon -- the current setup is a little confusing.

I'd just like to add that I fully agree. It's a confusing setup as it is now.

ah - thanks for the feedback, makes sense.

I will add that if you keep it as two buttons, the colors should be switched. Right now, the active button is black and the other is gray. But gray buttons say to me that they are disabled, so it looks like I can only hit the play button when it's already playing.

This was my interpretation too. Had to click it a few times to figure out what was going on.

I like it, but why not use something like htop? [1]

Perhaps it's not entirely comparable, but you do not need to open any extra ports or run any extra processes.

It looks really nice though! [1] https://en.wikipedia.org/wiki/Htop

I love htop too, but the great thing about scout_realtime is the couple minutes of context provided the charts.

Sometimes, nothing beats seeing a chart to quickly see what's going on.

That is true, charts are nice.

I actually mostly use glances [1]. I forgot to mention glances in my comment.

[1] https://pypi.python.org/pypi/Glances

I've found MenuMeters really useful (only available on Mac though)

http://www.ragingmenace.com/software/menumeters/index.html

It's always instantly clear when something like iTunes or a Chrome helper starts eating up 105% CPU, and provides quick access to force quitting it. :)

Screenshot: http://i.imgur.com/sEtkR9p.png

flexd, Htop or conky? http://conky.sourceforge.net/

acl...nice job. Your site looks nice and works nice.

I've never actually used conky. I'd never see it, I rarely notice my wallpaper because I never see my desktop.

I've recently released a similar tool for Python. Definitely not as pretty but with a focus on providing a lot of details: https://github.com/Jahaja/psdash

The more the merrier! :)

Did you roll your own SVG chart lib for this? If not, mind sharing which one you're using? It's very nice.

If you were to make it so I can open a socket or websocket to it (perhaps on a second, internal port) and publish whatever data I want, that'd be all kinds of nifty. That is, make it so I can just start spraying numbers at ws://myhost:5556/Really%20Awesome%20Data and with that a nice auto-scaled chart magically appears in the dashboard.

Edit: Oh, I see a github ribbon. Maybe you'll see a pull request sometime soon...

Edit 2: Anyone wondering about my original question - the charts are built using the D3 project.

http://d3js.org/

I've been searching for something like that for a while now. A metrics web framework that consumes and renders json/edn would be interesting.

I've been using shopify/dashing for metrics display:

http://shopify.github.io/dashing/

You can post JSON directly to each widget. It's also based on Sinatra.

Someone already created a widget for Rickshaw graphs, I suppose it could be adapted to use D3.js.

Our source for the charts is here: https://github.com/scoutapp/scout_realtime/blob/master/lib/s...

It's not yet in a state for plug-and-play usage in other projects. If you're looking to rollout smooth-scrolling charts quickly, checkout http://smoothiecharts.org/.

I just built this yesterday: http://blog.vivekpanyam.com/plotter-plot-anything/. Inspired by Tasseo and Scout Realtime

Cheers! If this were a "Show HN" I'd be happy to upvote! Edit: I see now that it kinda/sorta was, and I missed the opportunity.

Noob to Ruby here...any thoughts on why gem_original_require is screwing me up? Thanks for the cool looking tool!

/usr/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require': no such file to load -- json (LoadError) from /usr/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb:36:in `require' from /var/lib/gems/1.8/gems/scout_realtime-1.0.1/lib/scout_realtime.rb:23 from /var/lib/gems/1.8/gems/scout_realtime-1.0.1/bin/scout_realtime:4:in `load' from /var/lib/gems/1.8/gems/scout_realtime-1.0.1/bin/scout_realtime:4 from /usr/local/bin/scout_realtime:19:in `load' from /usr/local/bin/scout_realtime:19

$ gem install json

$ scout_realtime

You're welcome.

Looks nice, but it slowly turns my laptop into a fireball and pegs an entire core (older laptop).

Sorry. Just FYI, we've clocked the CPU usage of the scout_realtime daemon at 1% on an Intel Xeon 2.40GHz CPU.

I think we're talking about the browser side of things. I haven't investigated why, but it pegs the cpu on my macbook pro in Chrome, Safari, and Firefox.

It seems pretty light on the server side.

Ah thanks - I haven't seen this yet my laptop. We'll keep an eye on it.

It is burning my CPU (browser side, not server side) http://i.imgur.com/NgXi4LG.png The author should provide configuration so it does not get data from stats.json every second.

And I should add that this thing is awesome! It wouldn't install for me unless I was root, but that might be a ruby configuration thing, since this is also the first ruby app that I've ever touched. It would be nice to have an option to have scout_realtime only listen on 127.0.0.1, so (as someone else already mentioned) we could just proxy to it with another web server and then wrap controls around that.

With my 10 minutes of poking around in the scout_realtime source and a bit of googling, I think the option would go here, in main.rb:

   server = WEBrick::HTTPServer.new(:Port => 5555, :AccessLog => [])

I just tried out the main product and while it's easy to setup i ran into snags with the plugins right away. Both redis and postgres (the first two i tried) failed to install and it took a bit of searching to figure out they had their own dependencies. When trying to install those dependencies i ran into issues with compatibility for a fresh ruby install.

Maybe you should concentrate on fixing your own dependency issues before you start pounding on Nagios (https://scoutapp.com/info/nagios_alternative) about the exact same issue.

Looks bloody nice, got it up and running http://www.coldfusion10.com/installing-scout-realtime-on-cen...

This looks nice but that /s refresh it might cause issues, maybe add a setting so refresh can be set by user?

Shameless plug - If anyone is looking for a python/django alternative with refresh settings and remote access to the output data as json take a look at pyDash : https://github.com/k3oni/pydash .

Posted about it a while ago https://news.ycombinator.com/item?id=7224710 .

I'm working on a similar project, called dtop, check it out at https://github.com/ddierickx/dtop!

This is wonderful: No runtime required, minimal footprint, and almost no overhead when UI is idle.

Cool idea! The interface is a tad heavy (for me) though, the fans of my laptop spun up.

I noticed that by looking at the memory usage of the ssh daemon, one can determine how many people are connected with ssh. Every open connection (even if you're just idling at the login phase) adds around three to five mb to memory use. I wonder what other information might be unintentionally relayed through these metrics.

I'm glad to see the sparklines. I'm worried that the CPU sparklines in particular are likely to mislead due to the lack of a common vertical axis scale. I suspect Tufte might advise two graphs: one with a fixed 0…100 axis scale, one "zoomed". The former would help you compare CPU history between apps, and sport a shaded background region to indicate the range of the latter.

Good points. Thanks!

Regarding "What operating systems are supported?" and the answer of "scout_realtime relies heavily on the proc file system to fetch metrics. procfs is supported on most Linux-based distributions with the exception of OSX and Debian."

procfs is available on my Debian 7 servers, so scout_realtime just installed and runs fine on the few that I've tried it on.

And OS X is hardly a "linux distribution" :)

The web page makes firefox gobbles more and more RAM until it crashes. You've found a firefox bug :) (FF 27.1 Linux).

Any way of enforcing HTTPS and login/password auth? I would love to be able to use this on a set of public servers...

A simple proxy through nginx should do the trick.

Looks good. If you want to monitor network throughput on the CLI, use nload. It's great.

Does this solve the same problem that new relic does? If not, what is the difference?

Anyone with a link to a live demo?

http://scoutapp.com:5555/

kills the cpu on my mac with Chrome Version 34.0.1847.11 beta

how to view multiple servers on one screen?

Reply on news.ycombinator.com