Sunday, February 3, 2013

Splunk Howto - Splunk for Netfilter Iptables, get a great Iptables Multi-host frontend with Splunk! (Iptables log parser)







*** Updated June 14, 2013  ***

Current Version = 2.04

Major changes:
14/06/2013 - Details completion
03/06/2013 - New major release V2.0
29/05/2013 - Typo correction


To install this addon, follow this link on Splunkbase or install it through the standard Splunk application process search online: 
http://splunk-base.splunk.com/apps/74262/splunk-for-netfilter-iptables


Splunk pre-requirements:

Ensure to install requirements Splunk addons:



Splunk For Netfilter Iptables:


Home application page:

Centralized Home page including Real time alert trend and majors stats:


Page Details:

1. Realtime view menu with statically defined time window

Access to following views in Realtime data analysis mode with major time window:
  • Activity Overview: Complete analysis of Iptables activity with charts and stats
  • Google Map: World map overview of identified origin connections

2. View menu with TimeRange button access

Access to following views:
  • Activity Overview: Complete analysis of Iptables activity with charts and stats
  • Google Map: World map overview of identified origin connections
  • Event Search: Event search interface using various filters

3. Quick Activity Overview

  • Total number of connections denied in Realtime mode with a 24 hours time window
  • Date and hour of the first event indexed in Iptables index
  • Date and hour of the last event indexed in Iptables index


4. Alert trend:

Alert trend of denied connection activity in Realtime mode using a 24 hours time window: (Area chart)



5. Top Offenser stats:


Major stats about Top offenser in Realtime 24 hours time window mode:
  • Client IP: IP with the higher number of connections denied
  • Last Event Time: Date and hour of last connection occurence
  • Connections denied: Total number of connections denied for this host
  • Country origin: Identified country origin of this host (Unknown of unidentified)
  • Last destination IP: Last event destination IP
  • Last protocol: Last event protocol
  • Last source port: Last event source port
  • Last destination port: Last event destination port
  • Last service: Last event network service name (Unknown if non IANA reserved port)
  • Reporting server: Last event reporting server for this host

6. Last connections denied stats:




Major stats about last connection denied in Realtime 24 hours time window mode:
  • Client IP: IP of the last denied connection
  • Event Time: Date and hour of last connection denied
  • Interface: Logical interface name reported by Iptables
  • Country origin: Identified country origin of this host (Unknown of unidentified)
  • Destination IP: Destination IP of this event
  • Protocol: Protocol reported of this event
  • Source port: Source connection port
  • Destination port: Destination connection port
  • Service: Network service name (Unknown if non IANA reserved port)
  • Reporting server: Reporting server for this connection




Iptables Activity Overview Dashboard (Realtime/Timerange):




Dashboard details


1. Timerange overview:

  • Span Time value:
The span time value is dynamically defined using a Macro to get the best chart granularity, this is being used in chart command using "span=$Span$"

In Realtime mode, this is statically defined to 1 minute, in Timerange mode the range value can be automatically set from 5 minutes to several hours depending of the time range width.

  • Begin Time Analysis: Date and hour of the begining of the selected time range
  • End Time Analysis: Date and hour ot the end of the selected time range

2. Activity Summary:

This section presents various stats about more important informations of Iptables activity:


  • Number of connections denied: Total number of connections denied within the time range
  • Top offenser: Client IP with the highest number of connections denied
  • Denied for Top offenser: Total number of connections denied for the Top offenser host
  • Country origin for Top Offenser: Country origin for this host, Unknown if can't be identified
  • Top Protocol: Protocol most often attempted within the selected time range
  • Top Source port: Source port most often attempted
  • Top Destination port: Destination port most often attempted
  • Top Identified network service: Network service most often attempted
  • Top Destination IP: Destination IP most often reported
  • Top Country origin: Identified country most often reported
  • Top Reporting Host: Iptables host with the highest number of denied connections


3. Alert trend chart and peak load identification:



This sections shows:

- Alert trend by Iptables reporting host:

For Realtime, the span value (eg. span=) is statically defined in the XML hard code

For Timerange, the span value is dynamically defined by a macro (see Macro.conf) depending on the width of the time range itself, the goal here is to get the better chart granularity possible

- Peak load identification:

We identify here the peak load within the selected time range, how many connections were refused (drop and deny) and when.

Then this is being represented inside inside a gauge where the range (green, yellow, red range) will dynamically be defnied by a subsearch inside the global query.

These gauge range values will depend on the arithmetical mean result of denied connections for past 48 hours, the goal is to represent a potentil abnormal Iptables activity. (eg. being under attack)

The analysis result here should always equivalent between the chart and peak load. (both are cumulated results for all reporting hosts)



4. Last Events:



Last 100 events represented by major type of data, click on show result to export them or change the request.




4. TOP Source IP and Country Origin:

























- Top 10 pie chart and Top 100 data client IP with country origin identification

- Top 10 pie chart and Top 100 country origin of denied connections



5. TOP Network Services and Destination Port:



Data is being represented by:

- Top 10 pie chart and Top 100 data Network Services attempted:

Network services are identified whenever they are destination port known as reserved (for most of theme IANA ports reserved) using a csv input lookup.

See props.conf and transform.conf, Networking services are automatically defined for any event under the field called "Service", when the destination port is not known as reserved or standard port, the service name will defined under the value "Unknown".

- Top 10 pie chart and Top 100 destination port attempted with Network Service name identification



6. TOP Reporting Servers:



Data being represented by:

- Top 10 pie chart ad top 100 data Iptables Reporting Servers



Real-Time and Time-Range Google Maps view:


Real-Time and Time-Range Google Maps Dashboard, identify the source of connexion attempts




Event Search Interfaces:


Raw-Data and Charts Dedicated Event search interfaces:





And associated saved Searches...




System Dashboard, Iptables Index Activity:





Installation and utilization

Introduction

Installing and configuring Splunk is out of the scope of this post, installing Splunk is really easy and well done, in 10 minutes you'll be done ^^


As a brieve description, here is how Splunk for Iptables works:

- Configure Iptables to log dropped or rejected packets using Syslog
- Splunk collects our data and produces the IT intelligence

INFORMATION:

Please note that this application intends to analyse denied or dropped connections, any event containing following pattern will not be analysed by Splunk:

- "ACCEPT" OR "Accept" OR "ALLOW" OR "Allow"

Even if indexed by Splunk, if an event contains one of these patterns, it will be expected to be tagged as an accepted connection. If you need to adapt this general configuration to your own situation, please create a local Macro.conf containing the macro customized to your needs:

If you need custom settings, create your local Macro.conf:

$SPLUNK_HOME/etc/apps/iptables/local/Macro.conf:
### Iptables sourcetype
[iptables_src]
definition = sourcetype="iptables_source" NOT "ACCEPT" OR "Accept" OR "ALLOW" OR "Allow"
iseval = 0
Save your local/Macro.conf file and restart Splunk.

Installation and configuration will be done in a few steps:

SUMMARY OF STEPS:

Iptables / Syslog Configurations steps:

1. Set each Iptables reporting host to log events using SYSLOG

2. Configure SYSLOG to trap these events and put them in a dedicated log file of your choice

3. Test your Iptables / Syslog configuration to ensure events are being sucessfully trapped by your centralized Syslog

Splunk configuration steps:

4. Add Input source log file(s) in Splunk corresponding to the log file where your Syslog is trapping events

7. Open Splunk for Iptables and observe the magical power of Splunk ^^





Part 1 : Configuration of Iptables and Syslog

1. Set each Iptables reporting host to log dropped or rejected packets

Configuring Iptables is far away from the scope of this guide, the only thing required is to configure Iptables to log inbound dropped and rejected packets. (by default, Iptables logs its events to Syslog)

Iptables shall use a prefix pattern to log, this will be used first to manually recognize Iptables events in main syslog file, and then it shall be used to catched these events into a dedicated log file. (not obligatory but recommended)


Part 2 : Configuration of Syslog

In 2 steps:

- if you want to manage different Iptables reporting hosts servers from Splunk, then read the Multiple Iptables client configuration note
- If you just have one host to manage (Iptables and Splunk are installed in the same host), then just follow the common configuration section


MULTIPLE IPTABLES CLIENT CONFIGURATION NOTE: Remote and centralized Syslog configuration:

Configuring Syslog to send events from a Syslog host to a a remote Syslog server is out of the scope of this guide.
Therefore, if you want to send Iptables events of different hosts, you can choose between different solutions, as:

- Sending events using Syslog to a remote centralized Syslog
- Sending events from local log file using Splunk forwarder module
- Others (homemade scripts...)

I would recommend using "Rsyslog" (Enhanced Syslog version that comes with many modern Linux OS) to achieve this, which is indeed easy enough.

Here is in 2 steps a quick syslog centralized configuration: (remember to restart rsyslog after each modification)

1. In client rsyslog host, modify "/etc/rsyslog.conf" and add a section to send any events to your Syslog server: (adapt the example IP) 

*.* @192.168.1.254:514 


2. In syslog server configuration, create a configuration file that will catch any remote client Syslog events and put them into a dedicated per host log file:
Ensure your configuration name will be read after the iptables syslog config file. (see above, use numbered prefix)

Create "/etc/rsyslog.d/10-remotehosts.conf" with the following content: (Note: The iptables config we will create after will be called 08 to be read before this and intercept messages 

$template PerHostLog,"/var/log/%HOSTNAME%.log"
$template RemoteHostFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogfacility-text% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::space-cc,drop-last-lf%\n" 
:inputname, isequal, "imudp" ?PerHostLog;RemoteHostFileFormat
& ~


Finally, achieve the rest to the configuration above to be able to intercept remote Syslog client events containing the iptables event tag and put them in a dedicated log file for all hosts.


COMMON CONFIGURATION for Single and Multiple Iptables installations: 

1. Set Syslog to trap iptables events to a dedicated logfile

This configuration part will depend on your system and needs, i recommend the use of "rsyslog"

The goal is to configure syslog to trap any event containing a key word set as the iptables prefix into a dedicated log file

For example, with UFW you will have "[UFW BLOCK]" as a prefix.

If you set manually configure Iptables, just choose and report your log prefix. (eg. example "DROP(wan)"


In Debian/Ubuntu systems for example, create an rsyslog configuration file, example:

Create "/etc/rsyslog.d/08-iptables.conf" with the following content: (adapt with your log prefix) 

:msg, contains, "DROP(wan)" /var/log/iptables.log
& ~


Restart rsyslog to take effect.


Part 3 : Configuration of Splunk (the easy part!)

Configure input file using Splunk Manager interface:

Go to "manager", "Data Input", and configure MANUALLY a new input file pointing to your iptables log file.

With settings by part configuration:

Host:

You can let the default settings, it does not mind as we don't use it to recognize the iptables reporting server.

Source type:

- Set the source Type: Manual
- Source type: iptables_source

Index:

- Set the destination Index: iptables_index


Configure input manually:


You can also add your input file manually, create a new file in "${APP_HOME}/local/inputs.conf" with the following content: (adapt to your case)

[monitor:///var/log/remote-hosts/iptables_myserver.log]
disabled = 0
followTail = 0
index = iptables_index
sourcetype = iptables_source

[monitor:///var/log/iptables_ufw.log]
disabled = 0
followTail = 0
index = iptables_index
sourcetype = iptables_source


And restart Splunk.





Good news, you're done!!!
Just wait a few minutes to let Splunk get the content of your iptables log file, then go to the splunk application



41 comments:

  1. Thank you for this app. I am new to Splunk and was looking for something to parse my firewall logs from a Tomato router which uses iptables.

    I have this installed and it almost works. However when listing the top denied connections, it seems to show all. It's not filtering on just denied.

    I've found that if I edit the denied searches from:

    `iptables_index` `iptables_src` |

    to:

    `iptables_index` `iptables_src` DROP |

    Then it works as expected. Is there a better way?

    Below is an example of a "denied" syslog.

    Cheers,

    Drew

    Feb 1 00:05:52 blacksheep kernel: DROP IN=vlan1 OUT= MACSRC=00:0c:ef:98:40:00 MACDST=e0:91:f5:dc:95:f7 MACPROTO=0800 SRC=65.78.156.12 DST=69.62.230.77 LEN=52 TOS=0x00 PREC=0x00 TTL=123 ID=2536 DF PROTO=TCP SPT=59141 DPT=61083 SEQ=694582895 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B40103030201010402)
    Feb 1 00:05:53 blacksheep kernel: ACCEPT IN=vlan1 OUT=br0 SRC=65.78.156.12 DST=192.168.1.4 LEN=52 TOS=0x00 PREC=0x00 TTL=122 ID=2723 DF PROTO=TCP SPT=59142 DPT=443 SEQ=2773173856 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B40103030201010402)

    ReplyDelete
    Replies
    1. Hi Drew!

      Thanks for your feedback ^^

      Yes this in deed the expected behavior, i designed the application to only care with dropped or rejected connections. (which means only refused connection in log indexed)

      So as a requirement for dashboards to keep their plain meaning this application expects log to contain only dropped or refused connections and not connections allowed, moreover than keeping logs of allowed connections is not really useful in normal operation and may load the system with poor interest...

      Still, you adaptation if works is correct in your case, this will vary with the way your system identifies a dropped or allowed connection.

      You will have to adapt the base search in all saved searches into the config file savedsearches.conf.

      I didn't wanted to include the difference between the 2 cases inside the application as this may completely be different from one system to an other. (at the contrary of the rest of logs which will always be the same kind of)

      I have various improvements to do with this application and will update it soon, i'll try to see if i can integer this ^^

      Delete
    2. Hi Guilhem,

      Thank you for your reply. I am interested in both allowed and denied log entries. Maybe even more so in allowed because those are the connections that were not stopped by the firewall. If something is being allowed that should not, then I want to know about it. :)

      I will be interested in learning about Splunk and what it can do. For now I will just use your app for looking at denied connections and see if there is another app that is designed for reviewing allowed connections.

      Thank you for the tip. I'm so new I didn't even know about the savedsearches.conf file. I see my overrides were put in ./local/savedsearches.conf. Interesting to learn how this works.

      Cheers,

      Drew

      Delete
    3. Hi Drew,

      You're welcome ^^ You can off course modify yourself any search preset or Dashboard, if you do it through the interface it will in deed modify or create various config files in the "local" directory, and will overwrite rules set in default dir.

      Take a look at the source code search preset in savedsearches.conf and associated Dashboard files, you will easily learn how to modify them to your needs :-) (and still can revert in case of issue)

      Regards,

      Guilhem

      Delete
  2. I needed an PAL to NTSC Video converter for my American TV (LCD) I bought at Best Buy and saw my choice at World-Import.com website. I inquired from the Customer Service, paid for the item via bank transfer and got the item shipped to me at Port Harcourt Nigeria via UPS-EMS within 2 weeks. The converter is of amd socket robust and portable and perfectly serves my purpose. I am happy with the product and its fast delivery and can now view all available TV channels in Nigeria. I would want to do business again and again with World-Import. Thank you Victor for your good customer relations and great service!

    ReplyDelete
  3. You know the way you play around with words is simply phenomenal. This is such a well-crafted post. If you are among the students who are looking for fine quality and high scoring assignments developed by Ph.D. or Master’s qualified subject-specific from your background, the Dissertation Writing Service by the MyAssignmentHelpAU platform is the ultimate solution for you. You can easily get unmatched dissertation help for 100+ subject areas by paying a reasonable amount from your pocket.

    ReplyDelete
  4. Really its very informative post thanks for sharing this good stuff. If you are looking for a reliable name in the business who can handle all your thesis writing help experts then all you need to do is to connect with MyAssignmentHelpAU.

    ReplyDelete
  5. Hello There. I found your blog using Google. This is a very well written article . I will be sure to bookmark it and come back to read more of your useful information. Students who are looking for Dissertation Help United Kingdom are more likely to order their projects through us, because they know what quality they will be getting. Thanks for the post.

    ReplyDelete
  6. The dental insurance market is segmented on the basis of coverage, procedure type, demographics, end user, and region. In terms of coverage, it is segmented into Dental Preferred Provider Organizations (DPPO), Dental Health Maintenance Organizations (DHMO), Dental Indemnity Plans, and others. Based on procedure type, it is segmented into major, basic, and preventive. On the basis of demographics, it is segmented into senior citizens, adults, and minors. By end user, it is bifurcated into individuals and corporates. Region wise, the market is analyzed across North America, Europe, Asia-Pacific, and LAMEA.

    ReplyDelete
  7. Sleepwear is apparel designed to be worn while sleeping. The growth of the sleepwear market is attributed to the dynamic fashion trend and its influence on consumers. Consumers are eager to grasp the upcoming trend and demonstrate their fashionable lifestyle. This race to cope up with the changing fashion trend is anticipated to significantly influence consumers to purchase the latest fashion designs, especially worn by celebrities, and on the other hand also subtly influence to discard phased out apparel. In the present era, fashion trends have become bolder, which pushes consumers to flaunt their innerwear as casual loungewear or even sleepwear, thus propelling the growth of the sleepwear market. Furthermore, the rise in demand for trendy and fashionable nightwear has encouraged companies to launch new products to cater to the increasing needs of customers.

    ReplyDelete
  8. What an amazing blog and very well written. It solved all my queries perfectly. vice chancellor scholarship australia

    ReplyDelete
  9. If you are you searching Data Networking Cabling. then matoto technologies offer best services by Professional experts are highly skilled and they know how to deal with IT-related things. They will surely provide one of the best strategies and services to fulfill your specific services.

    ReplyDelete
  10. This blog is what im exactly looking for. Great! and Thanks to you. 토토

    ReplyDelete
  11. I am overwhelmed by your post with such a nice topic. Usually I visit your site and get updated through the information you include but today’s blog would be the most appreciable. 카지노

    ReplyDelete
  12. Great nice post !! keep writing or exploring we are learning from your posts. Here I would like to share with you my review on Assignment Help Qatar. Definitely, I would like to give them a full rating because they deliver my content at the given time even I need more pages they provide me with the same cost . They do not charge extra money to me. Sometimes they offer 50% discount on accounting subject which very beneficial for me and I purchase it. So I get more time to practice with other subjects and more explore it.

    ReplyDelete
  13. Major Private Toto refers to companies that share high cost sharing and events where there are a few restrictions when using the Toto site. Companies that participate in the work of the best oz's manufacturers and are responsible for real-time safety as a so-called safe playground or large sports venue at Totosite. 토토사이트 안전 토토사이트 추천 뱃사공 안전놀이터

    ReplyDelete
  14. Are you Worried to Know Intel 10th Gen Core i5 10500 Processor in bd Price in Bangladesh? Let’s discuss the Intel 10th Gen Core i5 10500 Processor. Model – Intel Core i5 10500, Code-Name – Comet Lake, CPU Cooler – PCG 2015C, Generation – 10th Gen., Base Frequency – 3.10 GHz, Turbo Frequency Max. – 4.50 GHz, Core – 6, Threads – 12, Smart Cache – 12 MB, Bus Speed – 8 GT/s, TDP – 65 W, Lithography – 14 nm, Memory Max. – 128 GB, Memory Type – DDR4-2666, Memory Chanels – 2, Processor Graphics – Intel UHD Graphics 630, Graphics Base Frequency – 350 MHz, Graphics Dynamic Frequency Max. – 1.15 GHz, Graphics Video Memory Max. – 64 GB, Graphics Resolution Max.

    ReplyDelete
  15. Keep up the good work , I read few posts on this web site and I conceive that your blog is very interesting and has sets of fantastic information.
    사설토토

    ReplyDelete
  16. 스포츠토토


    안전놀이터


    섯다족보



    To a great degree wonderful and entrancing post. I was hunting down this kind of information and had a great time examining this one.

    ReplyDelete
  17. Ask us Write my essay in Malaysia here at Malaysiaassignmenthelp.com and get your assignments completed by the expert academic writers just by a very nominal expense.

    ReplyDelete
  18. Exceptionally fantastic and captivating content. I enjoyed looking at this information because I was looking for it.
    best spine doctor in Delhi

    ReplyDelete
  19. Upto 70% off on Anarkalis, get 2in1 Heavy Net Zari Embroidered and Stonework Slit Anarkali from Laakha

    ReplyDelete
  20. Buy fitness equipment from the best online shop , SportFitzone in the USA. Also Enjoy Upto 50% Off Price on All Types of Gym or exercise equipment from SportFitzone in USA

    ReplyDelete
  21. Regards, thank you very much for sharing this IT related blog among all of us, people have got a lot of information from this blog. To avail more information or IT Support, Data Networking Cabling Services, connect with Matoto Technologies

    ReplyDelete
  22. Do you want to know about Graphic Design? Lets cheak our website and Collect your information

    ReplyDelete
  23. "I can't recommend this site enough! The articles are well-written and insightful, covering a wide range of topics.

    ReplyDelete
  24. Do you want to Improve your home? Lets cheak our website and Select which item do you want for.

    ReplyDelete
  25. Do you want to know about any facts information. Lets visit our website and find what you need.

    ReplyDelete
  26. Experience top-tier SEO services with Barakah Marketing Agency! Our expert team leverages cutting-edge strategies to enhance your online visibility and drive organic traffic. From keyword optimization to backlink building, we ensure your website ranks higher on search engines, attracting more potential customers. Partner with Barakah Marketing Agency and unlock the power of SEO for unprecedented business growth.

    ReplyDelete
  27. Golfing tee shirts make me feel like I'm part of an exclusive club of golf enthusiasts.Do you want to get offer price on Golfing tee shirts? Lets cheak our I've discovered some unique and stylish golfing tee shirts that have become my favorites.website and Select which item do you want for.

    ReplyDelete

  28. Need assistance with fixing your garage door in Laurel, MD? Look no further! We specialize in providing top-notch garage door repair services tailored to meet your needs. Our team of skilled technicians is well-equipped to handle any issue, whether it's a broken spring, malfunctioning opener, or damaged panels.

    With our expertise and attention to detail, we ensure that your garage door is restored to its optimal functionality in no time. Don't let a faulty garage door disrupt your daily routine - contact us today for reliable and efficient Garage door repair laurel md services.

    ReplyDelete

Please feel free to comment ^^