Friday, March 3, 2017

Integrating Elasticsearch and Logstash with NetApp Harvest and Grafana : Part 3

Forwarding NetApp's Syslog to Logstash

Hopefully your setup of ES and LS (Part 2 of this series) was a success.  If everything is up and listening then let's start sending some logs to your server.

Configuring CDOT v. 8.2, 8.3

Pushing your NetApp logs to your ES/LS server is real easy.  Lets start with CDOT 8.3 and 8.2 below.

You first need to create a destination name for your logs and tell OnTap where to send your them:
 event dest create -name netapplog -syslog yourLSservername  
You then need to route which messages you want to send to your new destination.  Be careful!  Your first instinct will to send everything!
 event route add-destinations -destinations netapplog -message *  
If your cluster(s) has a good number of nodes or the workload is pretty heavy then you will most likely drown in logs.  Most of which you don't need.  If you don't mind or you have a fairly small cluster that you are just testing on then you should be okay.  This will, at the least, give you an idea of what to expect and you can tailor what you send based on what you see or prefer.
Remember that CDOT has over 7000 different messages that can be triggered.  Maybe you or your customer are only interested in specific events. I don't have a list of all of them but you could run this command to view them.
 event route show -messagename *  
Say you are only interested in vserver messages:
 event route add-destinations -dest netapplog -message vserver*  

Doing a quick search of what that includes shows me this:
NetApp syslog "vserver" messages
My guess is that you or your customer are going to change what you want to see quite a bit.  To get an idea of what you may want to configure just ssh to your cluster and run "event log show".  Pick and choose what you want using the EMS_Identifier.

More Info:
8.3 Event Route Add-Destinations Info
8.2 Event Route Add-Destinations Info

Configure CDOT v 9.1

9.1 is basically the same except you have one more step.  You have to create a filter first then add a rule to the filter.  Then you create your destination then route the filter to it.  Command syntax looks like this:
 event filter create -filter-name netapplogs  
 event filter rule add -filter-name netapplogs -type include -message-name *  
 event notification destination create -name yourserver -syslog <yourserver-ip>  
 event notification create -filter-name netapplogs -destinations yourserver  

Your cluster should now be sending syslog messages to your server.  Let's verify!  Log into your server and run this command.
 curl -XGET "http://localhost:9200/_cat/indices?v"  
If you see an index then it looks to be working!
Working Index
Now let's see if our grok is working:
 curl -XGET "http://localhost:9200/logstash-2017.03.02/_search?pretty"  
Your output should look like this.  Notice all the fields and tags that our grok filter created!
Search Index Output
If this is what you see then you are done!  My next port (Part 4) will explain how to set up our new Elasticsearch server as a datasource on Grafana and we'll start building some dashboards!

Enjoy!

13 comments:

  1. Hi James, thank you for those great articles.
    Although you mentioned that most people will probably only be interested in a few message types, I was wondering if you have some sort of default list that you can reccomend? Many thanks!

    ReplyDelete
    Replies
    1. Thanks for reading! Sorry it took me so long to get back to you. Run this command "event status show". This will list all the events coming out of your cluster and give you an idea of what's happening. Pick and choose which ones you really want to track. I also recommend tracking emergency events out of the box, "event route -add-destinations {-severity EMERGENCY} -destinations ". Critical too.

      If you want to see what is in a severity level you can run "event route show -severity EMERGENCY" or "event route show -severity CRITICAL" etc...

      wafl.vol* and wafl.vvol.* are good ones to add too. It's all a matter of preference.

      Delete
  2. Hi,
    is part 4 coming? Great article so far.

    ReplyDelete
    Replies
    1. Part 4 has just been posted. I apologize for taking so long.

      Delete
  3. Is part 4 coming ?

    ReplyDelete
    Replies
    1. Part 4 has just been posted. I apologize for taking so long.

      Delete
  4. Still waiting on part 4.
    Does anyone has built already some sample dashboards for Grafana with included Elasticsearch queries?

    ReplyDelete
    Replies
    1. Part 4 has just been posted. I apologize for taking so long. I can put some together for you if you still need some.

      Delete
  5. I would be happy if you could write the parallels commands for it in ONTAP9.1? Its asking me about three options of alternative commands for everything after EVENT...

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. Prepare for Isaca CISM exam with our preparation material with full confidence. We offer you 100% real Certified Information Security Manager Isaca CISM exam dumps for your better results. Prepare4Test’s CISM pdf dumps are verified by Isaca Gurus.

    ReplyDelete
  8. 100% Pass Guarantee is offered by us after COBIT 5 Foundation COBIT5 exam preparation, with Prepare4Test’s exam dumps. You can pass the COBIT5 exam in the first go with good marks, and it will be easy for you to attempt all COBIT5 questions.

    ReplyDelete
  9. Prepare for ISTQB ATTA exam with our preparation material with full confidence. We offer you 100% real ISTQB Advanced Technical Test Analyst ISTQB ATTA exam dumps for your better results. Prepare4Test’s ATTA pdf dumps are verified by ISTQB Gurus.

    ReplyDelete