Justniffer

On 2010年09月16日, in linux, by netoearth

justniffer is a tcp packet sniffer. It can log network traffic in a customizable way. can simulate web server logs as the Apache access_log. may include information such as response times. useful to troubleshoot performance issues. It can also capture http traffic content (html, javascript, css, images, sounds, etc..) And save it in a directory

Main differences from other sniffers

Most of the sniffers are divided into two categories, packet an text sniffers. Both suffer from incompleteness of information that may be collected and analyzed

  • Packet sniffers collect too much data, such as packet headers details, and they make easy to analize low level network problem (such as tcp retrasmissions, or ip fragmentation) but make hard-working and time wasting to analize more high level details ( such as content correctness, keep-alive issues, connection timeouts, response time, etc)
  • Textmode sniffers usually rebuild TCP stream but cannot collect low level information such as timestamps. They often rebuild the tcp flow in a too simplistic way and fail when dealing with complex TCP/IP issues (reordering, retransmission, reassemlbying, etc). TCP reassembling and reordering is a complex exercise, and require a deep knowledge of TCP/IP protocol and long experience in the field. Usually,they are useful for pure grabbing content from network traffic.

Justniffer was born to help in toubleshooting perfomance in network tcp based services : HTTP, JDBC, RTSP, SIP, SMTP, IMAP, POP, LDAP, etc.
It can collect low and hight level protocol and performance info reconstructing the tcp flow in a reliable way using portions of the Linux Kernel code. Precisely, it uses a slightly modified version of the libnids libraries that already include a modified version of linux kernel code in a more reusable way.
You can extend traffic analisys with external scripts (bash, python, or any executable). An example is provided: justniffer-grab-http-traffic script use justniffer to saves files (images, text, html pages, javascript, flash, video, etc) captured from HTTP traffic.
Justniffer can generate logs in a customizable way. For example it can mimic the apache access_log

Justniffer features summary
TCP flow rebuild very reliable: it can reorder, reassemble tcp segments and ip fragments using portions of the Linux kernel code
Logging text mode: can be customized
Extensibility by any executable, such as bash, python, perl scripts, ELF executable, etc.
Performance measurement it can collect many information on performances: connection time, close time, request time , response time, close time, etc.

Examples

Example 1

Retrieving http network traffic and storing it in access_log format
justniffer -i eth0
192.168.2.2 – - [15/Apr/2009:17:19:57 +0200] “GET /sflogo.php?group_id=205860&type=2 HTTP/1.1″ 200 0 “” “Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)”
192.168.2.2 – - [15/Apr/2009:17:20:18 +0200] “GET /search?q=subversion+tagging&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:unofficial&client=firefox-a HTTP/1.1″ 200 0 “” “Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)”
192.168.2.2 – - [15/Apr/2009:17:20:07 +0200] “GET /sflogo.php?group_id=205860&type=2 HTTP/1.1″ 200 0 “http://justniffer.sourceforge.net/” “Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)”
192.168.2.2 – - [15/Apr/2009:17:20:18 +0200] “GET /csi?v=3&s=web&action=&tran=undefined&ei=MvvlSdjOEciRsAbY0rGpCw&e=19592,20292&rt=prt.175,xjs.557,ol.558 HTTP/1.1″ 204 0 “http://www.google.it/search?q=subversion+tagging&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:unofficial&client=firefox-a” “Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)”
192.168.2.2 – - [15/Apr/2009:17:20:07 +0200] “GET / HTTP/1.1″ 200 0 “” “Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)”

Example 2

We can append fields. For example http response time (see man page for a complete keyword list)
justniffer -i eth0 -a ” %response.time”
192.168.2.5 – - [22/Apr/2009:22:27:36 +0200] “GET /sflogo.php?group_id=205860&type=2 HTTP/1.1″ 200 0 “http://justniffer.sourceforge.net/” “Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)” 0.427993
192.168.2.5 – - [22/Apr/2009:22:27:50 +0200] “GET /complete/search?output=firefox&client=firefox&hl=en-US&q=add+e HTTP/1.1″ 200 140 “” “Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)” 0.294897
192.168.2.5 – - [22/Apr/2009:22:27:51 +0200] “GET /complete/search?output=firefox&client=firefox&hl=en-US&q=add+a HTTP/1.1″ 200 128 “” “Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)” 0.266929
192.168.2.5 – - [22/Apr/2009:22:27:21 +0200] “GET /extern_js/f/CgJlbiswCjgVLCswDjgFLCswFjgJLCswFzgBLCswGDgDLCswITgWLCswJTjJiAEsKzAmOAQsKzAnOAAs/-wB3HvFrpXA.js HTTP/1.1″ 304 0 “http://www.google.com/search?q=gnusticker&hl=en&safe=off&start=20&sa=N” “Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)” 2.025879

Example 3

You can capture all tcp traffic (add -u or -x options to encode unprintable characters):
justniffer -i eth0 -r
GET /doc/maint-guide/ch-upload.en.html HTTP/1.1
Host: www.debian.org
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en,it;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: UTF-8,*
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.debian.org/doc/maint-guide/
If-Modified-Since: Wed, 22 Apr 2009 19:36:31 GMT
If-None-Match: "400d604-3014-46829e160adc0"
Cache-Control: max-age=0

HTTP/1.1 304 Not Modified
Date: Wed, 22 Apr 2009 20:38:51 GMT
Server: Apache
Connection: Keep-Alive
Keep-Alive: timeout=15, max=100
ETag: "400d604-3014-46829e160adc0"
Expires: Thu, 23 Apr 2009 20:38:51 GMT
Cache-Control: max-age=86400

Example 4

You can define a completely custom log format
justniffer -i eth0 -l ” %request.timestamp %source.ip %dest.ip %request.header.host %request.url”
06/28/09 13:30:48 192.168.2.2 72.14.221.118 i1.ytimg.com /vi/TjSk6CVN5LY/default.jpg
06/28/09 13:30:47 192.168.2.2 72.14.221.118 i2.ytimg.com /vi/iw_nzfm1Vts/default.jpg
06/28/09 13:30:47 192.168.2.2 216.34.181.71 static.sourceforge.net /css/phoneix/jquery.cluetip.php?secure=0
06/28/09 13:30:48 192.168.2.2 216.34.181.71 static.sourceforge.net /sfx.js
06/28/09 13:30:49 192.168.2.2 216.34.181.71 static.sourceforge.net /include/coremetrics/v40/eluminate.js
06/28/09 13:30:51 192.168.2.2 199.93.61.126 c.fsdn.com /sf/images/phoneix/grad_white_dual_100.png

Example 5

You can read from a capture file. NOTE: capture file must be performed with unlimited snaplen for catching whole packets. Justniffer can work only works on pcap files with whole packets.
tcpdump command example: tcpdump -w /tmp/file.cap -s 0 -i ath0
justniffer -f /file.cap

Example 6

Many keyword has paramaters for setting a more precise formatting:
justniffer -i eth0 -l “%request.timestamp %request.header.host %request.url %response.time”
06/28/09 13:39:40 static.sourceforge.net /css/phoneix/print.css?secure=0&20080417-1657 0.162620
you can specifing timestamp formatting
justniffer -i eth0 -l “%request.timestamp(%B %d %Y %H:%M:%S) %request.header.host %request.url %response.time”
June 28 2009 13:39:40 static.sourceforge.net /css/phoneix/print.css?secure=0&20080417-1657 0.162620
or to print the string NoHostFound if the %request.header.host cannot be valued
justniffer -i eth0 -l “%request.timestamp %request.header.host(NoHostFound) %request.url %response.time”
06/28/09 15:10:28 www.google.com /ig?hl=en 0.116146
06/28/09 15:10:28 NoHostFound /ig?hl=en 0.116146
many keywords have their own formatting string. A generic option (-n) can be used for setting a default “not found” string: a string that must replace the keyword if it cannot be valued. for example if request.header.[headername] is not found or if connection.time cannot be applicable. Anyway, if a “no found” string is provided as keyword parameter (for those that expect it), it will override the -n option
justniffer -i eth0 -l “%request.timestamp %request.header.host(NoHostFound) %request.header.host %request.url %response.time” -n N/A
06/28/09 15:10:28 www.google.com www.google.com /ig?hl=en 0.116146
06/28/09 15:10:28 NoHostFound N/A /ig?hl=en 0.116146

Example 7

the -p option let you to specify a tcpdump compatibile filter: “port 80 or port 8080″ capture only http traffic (usually using tcp port 80 and 8080)
justniffer -i eth0 -r -p “port 80 or port 8080″

Example 8

the -e option let you to specify an external executable (usually a script) to which , for every log, the output will be redirect to. If you want to perform complex extraction operations, you can write your own script that will receive from the standard input all content specified by the -l option. A complete ad useful example is provided with justniffer-grab-http-traffic
./justniffer -l “%response” -e ./myscript.sh -i ath0
#!/bin/bash
# myscript.sh
# example script (print all lines containing "href" string)

while read inputline
do
 anchors=`echo "$inputline" | grep href`
 if [ "$anchors" != "" ]; then
    echo $anchors;
 fi;
done

Example 9

justniffer -i eth0 -r -p “port 25″
capture smtp traffic (usually using tcp port 25)
220 plecno.com ESMTP Postfix (Ubuntu)

EHLO unknown.localnet
250-plecno.com
250-PIPELINING
250-SIZE
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN                

MAIL FROM:<oreste.notelli@plecno.com> SIZE=1079
RCPT TO:<oreste.notelli@gmail.com>
DATA
250 2.1.0 Ok
250 2.1.5 Ok
354 End data with <CR><LF>.<CR><LF>            

From: Oreste Notelli <oreste.notelli@plecno.com>
Organization: Plecno
To: oreste.notelli@gmail.com
Subject: test
Date: Wed, 22 Apr 2009 22:46:16 +0200
User-Agent: KMail/1.11.2 (Linux/2.6.27-8-generic; KDE/4.2.2; i686; ; )
MIME-Version: 1.0
Content-Type: multipart/alternative;
  boundary="Boundary-00=_ZI47J3FTNXn+25g"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200904222246.17292.oreste.notelli@plecno.com>

--Boundary-00=_ZI47J3FTNXn+25g
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit

test

--Boundary-00=_ZI47J3FTNXn+25g
Content-Type: text/html;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html40/strict.dtd">
<html>
<head>
<meta name="qrichtext" content="1" />
<style type="text/css">p, li { white-space: pre-wrap; }</style>
</head>
<body style=" font-family:'DejaVu Sans'; font-size:8pt; font-weight:400; font-style:normal;">
<p>
test
</p>
</body>
</html>
--Boundary-00=_ZI47J3FTNXn+25g--
.
250 2.0.0 Ok: queued as 33E7235C21A

QUIT
221 2.0.0 Bye
Tagged with:  

Comments are closed.