Integrating Elasticsearch and Logstash with NetApp Harvest and Grafana : Part 1

Digging even deeper...

Background

I've been using a great tool called NetApp Harvest for a few years now.  If you are familiar with Harvest and how it integrates with Grafana and Graphite then you know what this tool can provide you and/or your customers.  A colleague of mine at NetApp , Chris Madden (@dutchiechris on twitter) is the author and maintainer of NetApp Harvest and you can grab all the info you need on it from his blog called "it all begins with data".  One of the great things about Harvest is that it comes with  a ton of dashboards ready to install on Grafana!  Another colleague of mine, Yann Bizeul (@ybontap on twitter) built an "all-in-one" OVA template called the NAbox (formerly called ADVA).  The NAbox makes deploying Harvest/Grafana/Graphite very easy.  In minutes you can get it up and running and see beautiful metrics pouring out from your NetApp systems.  All the information you need to deploy this wonderful tool is located here https://nabox.tynsoe.org/.

NetApp Dashboard: Cluster Group

My customer has come to depend on this tool everyday.  They are a global company that uses various applications that demand performance and scalability.  Downtime to their clients is very expensive to them.  The NAbox helps them track down potential bottlenecks and various other performance issues.  Everything here is automated so there are a lot of moving parts going on around the world and the NAbox provides them one place to go for performance metrics on all of their storage.

 

I came up with an idea not to long ago about adding even more information to what my customer already has.  I was familiar with what Elasticsearch could do and I knew that Grafana had a datasource plugin for it so I thought "Wouldn't it be cool if I could use Elasticsearch to annotate current dashboards with logs from each system?".  I know folks have been using ELK (Elasticsearch/Logstash/Kibana) and Elasticsearch on Grafana for a long time so I'm not telling most of you anything new here.  What I have not seen out there is someone using the NAbox (designed specifically for NetApp storage) and integrating it with syslog event monitoring.

 

Here is a screenshot of an example:  you'll notice a spike in disk utilization on one of the aggregates below and at the bottom we see an event that was triggered around that same time.

Screenshot of NetApp Dashboard: Node with annotations

It goes with saying that my customer was thrilled with this idea.  Once they saw it they loved it even more.  Think about the amount of data you can now use to analyze your system's performance (or even prove that the root cause of an outage was not storage).  If you keep track of mirroring events you could see how they affect performance, especially if all your mirrors begin at the same time of day.  Ever wonder what happens to a raid group when a disk fails?  You can track and visualize that too!  In heavily automated environments, like mine, where clients have the ability to create and destroy Flexclones you can visualize something like this:

Flexclone tracking

 

The list goes on and on!  In part 2 of this blog series (coming very soon!) I'll show you where and how to start putting this together.

 

Happy monitoring!