Friday, January 5, 2018

Building a SEIM Dashboard with R, Jupyter, and Logstash/Elastic Search

Motivation:

I am disappointed with the dashboards offered by today's SEIMs.  SEIM dashboards offer limited data manipulation through immature, proprietary query languages and limited visualization options. Additionally, they tend to have proprietary data stores that limit expansion and evolution to what the vendor supports.  Maybe I'm spoiled by working in R and Rstudio for my analysis, but I think we can do better.

Plan:

This blog is mainly going to be technical steps vs a narrative.  It is also not the easiest solution.  The easiest solution would be to already have the ELK stack, install interact.io, R, the R libraries, and the R jupyter kernel on your favorite desktop, and connect.  That said, I'm going to walk through the more detailed approach below.  You can view the example notebook HERE.  Make sure to scroll down to the bottom where the figures are as it has a few long lists of fields.

Elastic search is becoming more common in security, (e.g. 1, e.g. 2).  Combine that with the elastic package for R, and that should bring all of the great R tools to our operational data.  Certainly we can create regular reports using Rmarkdown, but can we create a dashboard?  Turns out with Jupyter you can!  To test it out, I decided to stand up a Security Onion VM, install everything needed, and build a basic dashboard to demonstrate the concept.

Process:

Install security onion:

Security onion has an EXCELLENT install process.  Simply follow that.

Install R:


Added ‘deb https://mirrors.nics.utk.edu/cran/bin/linux/ubuntu trusty/‘ to packages list

sudo apt-get install r-base

sudo apt-get install r-base-dev

— based off r-project.org

Install R-studio (not really necessary but not a bad idea)


Downloaded r-studio package from R-studio and installed

Sudo apt-get install libjpeg62

sudo dpkg -I package.deb

Install Jupiter:


(https://www.digitalocean.com/community/tutorials/how-to-set-up-a-jupyter-notebook-to-run-ipython-on-ubuntu-16-04)

Sudo apt-get install python-pip

sudo pip install —upgrade pip (required to avoid errors)

sudo -H pip install jupyter 

Install Jupyterlab: (probably not necessary)


Sudo -H pip install jupyterlab

Sudo jupyter serverextension enable --py jupyterlab --sys-prefix

Install Jupiter dashboard

(https://github.com/jupyter/dashboards)

sudo -H pip install jupyter_dashboards

sudo -H pip install --upgrade six

Sudo jupyter dashboards quick-setup --sys-prefix 

Install R packages & Jupypter R kernel:

Sudo apt-get install libcurl4-openssl-dev

sudo apt-get install libxml2-dev

Start R

install.packages("devtools") # (to install other stuff)

install.packages(“elastic”) # talk to elastic search

install.packages(“tidyverse”) # makes R easier

install.packages("lubridate") # helps with working with dates

install.packages("ggthemes") # has good discrete color palettes

install.packages("viridis") # has great continuous colors

# https://github.com/IRkernel/IRkernel

devtools::install_github('IRkernel/IRkernel')

# or devtools::install_local('IRkernel-master.tar.gz')

IRkernel::installspec() # to register the kernel in the current R installation

quit() # leave. Answer ’n’ to the question “save workspace?”

Install nteract: (Not necessary)

(nteract.io)

Download the package

Sudo apt-get install libappindicator1 libdbusmenu-gtk4 libindicator7

sudo dpkg -i nteract_0.2.0_amd64.deb


Set up the notebook:

Rather than type this all out, you can download an example notebook.  In case you don't have an ES server populated with data, you can download this R data file which is a day of windows and linux server logs queried from ES from a blue vs red CTF.

I created the notebook using nteract.io so it is in a single order.  However, if you open it on the juypter server, you can use the dashboards plugin to place the cells where you want them in a dashboard.

Results:

A lot of time spent compiling.

No need to download R/jupyter stuff on security onion if elastic search is remotely reachable.

Elastic search is not intuitive to query.  Allowing people an 'easy mode' to generate queries would be significantly helpful.  the `ES()` function in the workblook is an attempt to do so.

It would be nice to be able to mix interactive and dashboard cells.

This brings MUCH more power for both analysis _and_ visualization to the dashboard.

This brings portability, maintainability (ipynb files can be opened anywhere that has the R/jupyter environment and can access elastic search.  They can also be forked, version controlled, etc.)

Future Work:

Need a way to have cells refresh every few minutes, likely a jupyter notebook plugin.

Interactive figures require interactive plotting tools such as Vega.  This would also bring the potential ability to stream data directly to the notebook.  It may even solve the ability to auto-refresh.

Conclusion:

In conclusion, you really don't want to roll-your-own-SEIM.  That said, if you already have ES (or another data store R can talk to) in your SEIM and want less lock-in/more analysis flexibility, R + Jupyter may be a fun way to get that extra little emph.  And hopefully in the future we'll see SEIM vendors supporting general data science tools (such as R or Python) in their query bars and figure grammars (ggplot, vega, vegalite), in their dashboards.

26 comments:

  1. I'm super excited that someone appears to be way farther along in this than me! Check it out at: https://github.com/Cyb3rWard0g/HELK

    ReplyDelete
    Replies
    1. Such an effective topic, also there is well deserving CIPD assignments is always available for students in achieving better academic results.

      Delete
  2. I would definitely thank the admin of this blog for sharing this information with us. Waiting for more updates from this blog admin. Melbourne CCTV Installers

    ReplyDelete
  3. I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you! Building surveillance camera system upgrade

    ReplyDelete
  4. This is my first time visit to your blog and I am very interested in the articles that you serve. Provide enough knowledge for me. Thank you for sharing useful and don't forget, keep sharing useful info: security camera installation

    ReplyDelete
  5. I’m going to read this. I’ll be sure to come back. thanks for sharing. and also This article gives the light in which we can observe the reality. this is very nice one and gives indepth information. thanks for this nice article... Serious Security Melbourne

    ReplyDelete
  6. I’m going to read this. I’ll be sure to come back. thanks for sharing. and also This article gives the light in which we can observe the reality. this is very nice one and gives indepth information. thanks for this nice article... Hikvision

    ReplyDelete
  7. The article was up to the point and described the information very effectively. Thanks to blog author for wonderful and informative post.
    Security Solution consultant

    ReplyDelete
  8. Thanks for sharing the post.. parents are worlds best person in each lives of individual..they need or must succeed to sustain needs of the family. CCTV camera

    ReplyDelete
  9. This is such a great resource that you are providing and you give it away for free. I love seeing blog that understand the value of providing a quality resource for free. Hikvision DS-2CD2385G1-I

    ReplyDelete
  10. Awesome and interesting article. Great things you've always shared with us. Thanks. Just continue composing this kind of post. CCTV Installers Melbourne

    ReplyDelete
  11. This outdoor security camera system provides clear colored images during the day and black and white images at night. This type facilitates nighttime surveillance with various resolutions and lens angles. best security cameras

    ReplyDelete
  12. If you are looking for more information about flat rate locksmith Las Vegas check that right away. Security Systems Melbourne

    ReplyDelete
  13. Great post, and great website. Thanks for the information! cctv camera

    ReplyDelete
  14. I totally agree with this article and I just want to say that this article is a very nice and very informative article.I will make sure to be reading your blog more. Opt for the home wireless cctv kits

    ReplyDelete
  15. Many structure execution issues can be followed to air spillage through the structure envelope. These issues go from high warming expenses and helpless temperature control in consumed spaces, to rain entrance and the decay different segments inside a structure get together. maintenance scheduling

    ReplyDelete
  16. Great write-up, I am a big believer in commenting on blogs to inform the blog writers know that they’ve added something worthwhile to the world wide web!.. Hikvision Acusense

    ReplyDelete
  17. Thank you because you have been willing to share information with us. we will always appreciate all you have done here because I know you are very concerned with our. Serious CCTV Melbourne

    ReplyDelete
  18. When collecting data, whether qualitative or quantitative, we can use several tools, including surveys, focus groups, interviews, and questionnaires. It depends on you what kind of tool you use to help organize the data. Dissertation Editing Service

    ReplyDelete
  19. To build a SEIM dashboard with R, Jupyter, and Logstash/Elasticsearch, you can start by setting up Logstash to collect and parse security event logs. Then, use Elasticsearch to store and index the data. Next, use R and Jupyter to analyze and visualize the SEIM data. If you need assistance with this task or think how i do my assignment, feel free to ask.

    ReplyDelete
  20. I am fascinated by the concept of building a SEIM Dashboard with R, Jupyter, and Logstash/Elastic Search. It's exciting to see how data analysis and visualization can be applied in such a dynamic field. On a different note, I recently came across some great deals for car sell online in Sharjah. If anyone is looking to buy or sell a car in the area, I highly recommend Expact car buyers checking it out!

    ReplyDelete
  21. Creating a SEIM dashboard is like assembling the perfect tech puzzle. It's all about precision and efficiency, just like hunting for the best deals on pizza. Both endeavors demand meticulous planning and execution. While one secures our digital fortress, the other ensures a delightful pizza feast without breaking the bank.

    ReplyDelete
  22. NY212 Pizza Pakistan consistently offers some of the best pizza deals in Karachi, making every bite both delightful and budget-friendly. If you're in the mood for quality and value, NY212 is the place to be!

    ReplyDelete