Category Archives: Project 7 – Network Analyzer Ovizart-NG

Network Analyzer Project Updates (Hao Ma) – Week 13 – More Examples

1. Test for a Trojan binary file:

$ sudo python ovizcli.py -i /Users/zqzas/Downloads/MyLogerMailEnd.exe -vt -o /tmp

$ {“scan_id”: “eb2ba9d47c3a3c0120738069bc146de637497b60ab0d4152e582d80c136f1d68-1379835752″, “sha1″: “c83478bc431e936f36919c59103bd6ba845c8060″, “resource”: “eb2ba9d47c3a3c0120738069bc146de637497b60ab0d4152e582d80c136f1d68″, “response_code”: 1, “sha256″: “eb2ba9d47c3a3c0120738069bc146de637497b60ab0d4152e582d80c136f1d68″, “permalink”: “https://www.virustotal.com/file/eb2ba9d47c3a3c0120738069bc146de637497b60ab0d4152e582d80c136f1d68/analysis/1379835752/”, “md5″: “7d867d6bd5fc3015a31fdfa121ba9187″, “verbose_msg”: “Scan request successfully queued, come back later for the report”}

 

Then pop out a web page showing the results in a table and saying that your request has been queued.

QQ20130922-1

 

You might go to the permalink for further information later.   Like https://www.virustotal.com/file/eb2ba9d47c3a3c0120738069bc146de637497b60ab0d4152e582d80c136f1d68/analysis/1379835752/

According to the VirusTotal, it is absolutely a malicious binary file most likely a Trojan exe.

 

1. Test for a malicious website:

http://xa.jjhh.com/

Issue:

    $ sudo python ovizcli.py -i http://xa.jjhh.com/ -vt -o /tmp

Then the report  page will pop out: saying its detected as malware site by Google Safebrowsing, Sophos, and Fortinet.

The jsunpack-n results can be checked by changing “vt” to js in the command:

  $ sudo python ovizcli.py -i http://xa.jjhh.com/ -js -o /tmp

 

 

 

 

 

 

Network Analyzer Project Updates (Hao Ma) – Week 12 – Testing Report

The Ovizart-ng is able to handle basically 4 types of input:

1. PCAP: use the core analyzer

2. URL: may call the extended analyzer like VirusTotal, and Jsunpack-n

3. binary file: VirusTotal and Cuckoo analyzer may handle

4. text file: (like html and javascript file): Jsunpack-n analyzer may handle.

HOWTO:

1. If you’d like to analyze a pcap, there’re 2 ways :

1). use cli tool of ovizart-ng:

Example:

$ sudo python ovizcli.py -i /Users/zqzas/Projects/ovizart-ng/test/pcap/test-http.pcap -o /tmp
I’m awesome
name: /Users/zqzas/Projects/ovizart-ng/test/pcap/test-http.pcap type: PCAP
Analysis Object{
id: 1
startTime: 2013-09-10 15:41:14.105801
user: <NoUserDefined>
config: <ovizconf.Config instance at 0x101b9a0e0>
status: FINISHED
data: [Data Object{
tags: {‘data_source': ‘PCAP’, ‘app_layer_protocol': ‘HTTP’, ‘attachments': [('_Websidan_index.html', 'regular file', None)], None: ['_Websidan_index.html']}
data: {‘stream': Stream Object {key: 6_10.1.1.101_3188_10.1.1.1_80, protocol: 6, srcIP: 10.1.1.101, srcPort: 3188, dstIP: 10.1.1.1, dstPort: 80, startTime: 1100903355.43, numberOfPacket: 14, pcapFile: /tmpanalysis_20130910_154114_105820/test-http.pcap/6_10.1.1.101_3188_10.1.1.1_80/6_10.1.1.101_3188_10.1.1.1_80.pcap}}
}, ……omitted

2) use interactive tool of ovizart-ng:

 

 

 

$cd shell/

$python ovizshell.py

(Cmd) set input = /Users/zqzas/Projects/ovizart-ng/test/pcap/test-http.pcap
(Cmd) set output = /tmp
(Cmd) show
{‘output': ‘/tmp’, ‘external_tool': ”, ‘verbose': ”, ‘input': ‘/Users/zqzas/Projects/ovizart-ng/test/pcap/test-http.pcap’}
(Cmd) start
name: /Users/zqzas/Projects/ovizart-ng/test/pcap/test-http.pcap type: PCAP
Analysis Object{
id: 1
startTime: 2013-09-10 15:20:15.713222
user: <NoUserDefined>
config: <ovizconf.Config instance at 0x101c99908>
status: FINISHED
data: ….omitted

 

2. To analyze a url:

(Cmd) set input = http://honeynet.org
(Cmd) set output = /tmp
(Cmd) set external_tool = -vt
(Cmd) start
name: http://honeynet.org type: URL
Virus-total analyzing …………………………
['http://honeynet.org']
——————-
{“permalink”: “https://www.virustotal.com/url/7547b57712941e07a6f9f786a6f311b534c94c0e2ba59126d7f1ef4ff24866e4/analysis/1377971788/”, “url”: “http://honeynet.org/”, “response_code”: 1, “scan_date”: “2013-08-31 17:56:28″, “scan_id”: “7547b57712941e07a6f9f786a6f311b534c94c0e2ba59126d7f1ef4ff24866e4-1377971788″,….omitted

set to another external analyzer, “jsunpack-n” :

(Cmd) set external_tool = -js
(Cmd) show
{‘output': ‘/tmp’, ‘external_tool': ‘-js’, ‘verbose': ”, ‘input': ‘http://honeynet.org’}
(Cmd) start
name: http://honeynet.org type: URL
Jsunpack-n analyzing …………………………

http://honeynet.org

!!! /Users/zqzas/Projects/ovizart-ng/analyzer/jsunpack_n/jsunpack-n-read-only
The key / has the following output in recursive mode
[nothing detected] /
info: [0] no JavaScript
file: stream_bf9b49684b9623595fbb8e12648d3d19ecb5c77c: 19 bytes
Note that none of the files are actually created since self.outdir is empty.
Instead, you could go through each url and look at the decodings that it creates
Looking at key /, has 1 files and 1 messages, that follow:
file type=stream, hash=bf9b49684b9623595fbb8e12648d3d19ecb5c77c, data=19 bytes
output message printable=1, impact=0, msg=[0] no JavaScript

Response:
[['The reports has been saved in /Users/zqzas/Projects/ovizart-ng/analyzer/jsunpack_n/jsunpack-n-read-only/log.'], []]

 

 

 

 

2. To analyze a binary:

1) VirusTotal:

(Cmd) set input = /Users/zqzas/Downloads/anyexe.exe
(Cmd) set output = /tmp
(Cmd) set external_tool = -vt
(Cmd) start
name: /Users/zqzas/Downloads/anyexe.exe type: BINARY
Virus-total analyzing …………………………

{“scan_id”: “209342a2755315c7cef091f4f56de0875ee9cafee73814c05faf5db1a3955ee4-1378802153″, “sha1″: “5d92013fe866395a1c5370192d9ad83e88328a64″, “resource”: “209342a2755315c7cef091f4f56de0875ee9cafee73814c05faf5db1a3955ee4″, “response_code”: 1, “sha256″: “209342a2755315c7cef091f4f56de0875ee9cafee73814c05faf5db1a3955ee4″, “permalink”: “https://www.virustotal.com/file/209342a2755315c7cef091f4f56de0875ee9cafee73814c05faf5db1a3955ee4/analysis/1378802153/”, “md5″: “fb086841437211545b5260209fa9ecf7″, “verbose_msg”: “Scan request successfully queued, come back later for the report“}

2)Cuckoo:

(Cmd) set input = /Users/zqzas/Downloads/anyexe.exe
(Cmd) set output = /tmp
(Cmd) set external_tool = -ck
(Cmd) start
name: /Users/zqzas/Downloads/anyexe.exe type: BINARY
Cuckoo analyzing …………………………
You may check the reports at: ( http://81.167.148.242:8090/tasks/view/202 ) after it’s available.

 

 

4. text file, a html file with js:

(Cmd) set input = /Users/zqzas/Projects/ovizart-ng/shell/report.html
(Cmd) set output = /tmp
(Cmd) set external_tool = -js
(Cmd) start
name: /Users/zqzas/Projects/ovizart-ng/shell/report.html type: PLAINTEXT
Jsunpack-n analyzing …………………………
/Users/zqzas/Projects/ovizart-ng/shell/report.html
!!! /Users/zqzas/Projects/ovizart-ng/analyzer/jsunpack_n/jsunpack-n-read-only
The key / has the following output in recursive mode
[nothing detected] /
info: [0] no JavaScript
file: stream_e4a62c83ace44261a545060c454a6c6fd3c677f1: 50 bytes
Note that none of the files are actually created since self.outdir is empty.
Instead, you could go through each url and look at the decodings that it creates
Looking at key /, has 1 files and 1 messages, that follow:
file type=stream, hash=e4a62c83ace44261a545060c454a6c6fd3c677f1, data=50 bytes
output message printable=1, impact=0, msg=[0] no JavaScript
Response:
[[], ['The reports has been saved in /Users/zqzas/Projects/ovizart-ng/analyzer/jsunpack_n/jsunpack-n-read-only/log.']]

Above are the cases of using interactive shell, which can be achieved by ovizcli.py equivalently as well.

 

 

 

Network Analyzer Project Updates (Hao Ma) – Week 11 – Interactive Shell Demonstration

As I mentioned last week, I was trying to complete interactive shell module of ovizart-ng this week.

Initially, I was implementing the command line shell all by myself. After my mentor Oguz suggested me to take a look at cmd module of python (http://docs.python.org/2/library/cmd.html, http://pymotw.com/2/cmd/#module-cmd), I found that cmd and readline are very helpful for our interactive shell of ovizart-ng because it’s convenient to achieve autocompletion and organize help information.

Now I have implemented the interactive shell using cmd and readline module. Some demos are following:

Introduction:

A interactive shell for ovizart-ng with the autocomplete(press tab) and help support
Example:
> set input = http://mal.site
> set output = /tmp
> set external_tool = -js
> show
> start

For the more info, you may also refer to ovizcli.py

———

(Cmd) help

Documented commands (type help <topic>):
========================================
EOF list reset set show start version

 

———

(Cmd) help set
Set the config data.
(Cmd) show
{‘output': ”, ‘external_tool': ”, ‘verbose': ”, ‘input': ”}
(Cmd) set input = /Users/zqzas/Downloads/ping.pcap
(Cmd) set output = /tmp
(Cmd) show
{‘output': ‘/tmp’, ‘external_tool': ”, ‘verbose': ”, ‘input': ‘/Users/zqzas/Download/ping.pcap’}
(Cmd) start

name: /Users/zqzas/Downloads/ping.pcap type: PCAP

Analysis Object{
id: 1
startTime: 2013-09-03 18:53:58.196572
user: <NoUserDefined>
config: <ovizconf.Config instance at 0x101c98998>
status: FINISHED
data: [Data Object{
tags: {'data_source': 'PCAP', 'app_layer_protocol': 'UNKNOWN'}
data: {'stream': Stream Object {key: 1_175.186.61.135_123.125.114.144, protocol: 1, srcIP: 175.186.61.135, srcPort: None, dstIP: 123.125.114.144, dstPort: None, startTime: 1371300147.29, numberOfPacket: 8, pcapFile: /tmpanalysis_20130903_185358_196603/ping.pcap/1_175.186.61.135_123.125.114.144/1_175.186.61.135_123.125.114.144.pcap}}
}, Data Object{
tags: {'data_source': 'PCAP', 'app_layer_protocol': 'UNKNOWN'}
data: {'stream': Stream Object {key: 1_175.186.61.135_10.8.8.8, protocol: 1, srcIP: 175.186.61.135, srcPort: None, dstIP: 10.8.8.8, dstPort: None, startTime: 1371300141.12, numberOfPacket: 8, pcapFile: /tmpanalysis_20130903_185358_196603/ping.pcap/1_175.186.61.135_10.8.8.8/1_175.186.61.135_10.8.8.8.pcap}}
}]
files: [{'numberOfStreams': 2, 'numberOfPacket': 16, 'filename': '/Users/zqzas/Downloads/ping.pcap'}]
}

and at the same time, report.html will be placed in the current folder.

 

 

Network Analyzer Project Updates (Hao Ma) – Week 10 – CLI Demonstration

Here a small demo around the CLI of ovizart-ng:

If you’re a fresh user and wanna try ovizart-ng at this moment, you may first download the source code from https://github.com/honeynet/ovizart-ng/tree/hao-devel. You’ll find ‘ovizcli.py’ in the directory and that ‘python ovizcli.py –help’ is very helpful.

Here is what you get:

QQ20130827-2

 

Next, you probably would like to have a first bite on the analyzers and I guess it’s a nice choice to start from VirusTotal:

Issue: “python ovizcli.py -i http://google.com -vt -o /tmp”

As the –help suggested, “-i” is for the input, “-vt” is for Virustotal.

The results are below:

QQ20130827-3

In the CLI, the display of results is not very friendly for human eyes which likes a json mess. But we have built the reporters which are able to transform json dictionary into html and pdf displayed in the tables of Bootstrap that might look nice.

Then, Jsunpackn and Cuckoo Sandbox are also worth a try. All together we have 3 external analyzers.

The internal analyzer is under development. More demos will follow.

By the way, besides CLI, we are busily developing the shell type interface as well. The demo of shell.py would be available next week.

 

Network Analyzer Project Updates (Hao Ma) – Week 9 – Reporter Module Sample

Recently, I added unit tests for HTML reporter and PDF reporter.
The unit test programs are to convert json:

‘{“permalink”: “https://www.virustotal.com/url/b618f0bfbd90176adfcbc2ee9854140f0739063d87807144e26283b0e44791f7/analysis/1370430938/”, “url”: “http://zqzas.com/”, “response_code”: 1, “scan_date”: “2013-06-05 11:15:38″, “scan_id”: “b618f0bfbd90176adfcbc2ee9854140f0739063d87807144e26283b0e44791f7-1370430938″, “verbose_msg”: “Scan finished, scan information embedded in this object”, “filescan_id”: null, “positives”: 0, “total”: 39, “scans”: “omitted…”}’

into readable HTML. Given HTML file, the PDF reporter is able to transform HTML to PDF then.

For example, you may simply run “unittest_html.py” under reporter folder then you will obtain a “report.html” in the same folder.

Network Analyzer Project Updates (Hao Ma) – Week 8

In the week 8, we entered the tasks for final term.
Two things were done:
1. Fix CLI
2. reporter modules

For the CLI, I fixed some bugs(sync my code with my teammate Gurcan), and added verbosity option into. Now it seems more complete.

For the reporter, I’ve written 2 modules: HTML reporter and PDF reporter.
First, the input data is either a json string or a dictionary (they are almost same thing). Then I will convert the data into inner representation (a dictionary) and embedded them into the HTML which came from a template. The HTML template feature is supported by Jinja2 (http://jinja.pocoo.org) that is very similar with the Django’s template system I am familiar with.
The HTML is under Bootstrap.

Now, we’ve already have reporter that converted results into HTML. Thus, we are able to get the PDF version by converting HTML to PDF. This can be achieved by xhtml2pdf (https://github.com/chrisglass/xhtml2pdf). By the way, there are bugs when xhtml2pdf works with Bootstrap, so I have to follow its errors info to edit bootstrap.css to make them compatible.

In the coming week, my task is shell.py (a shell type interface beyond the CLI).

Network Analyzer Project Updates (Hao Ma) – Week 7

This is mid-evaluation week. My mentor and I focused on our plan till the final. After Discussion, we decided to develop the shell type interface (interactive) as a awesome feature of our ovizart-ng.
The brief introduction of shell type interface in ovizart-ng is below:

>>> list datasources | taggers | analyzers | reporters > read smtp.pcap
>>> info [general | streams | stream id]
>>> split HTTP as pcap | file
>>> analyze with VT [stream-id] > view html | pdf
>>> split 100M

Another goal of following week is to implement the reporter module of ovizart-ng. The reporters should export the analysis reports as pdf or html files that are readable format for human.

The timeline to final:
Aug 6th – Aug 13th : Enhanced CLI and reporter module
Aug 13th – Aug 20th: Finish CLI and reporter. And start to write shell.py
Aug 20th – Aug 27th: Write shell.py
Aug 27th – Sep 3rd: Finish shell.py. Start to implement DNS analyzer
Sep 3rd – Sep 10th: Implement DNS analyzer
Sep 10th – Sep 17th: Finish everything properly. (after further discussion)
Sep 17th – Sep 24th: Get ready for final evaluation.

Network Analyzer Project Updates (Hao Ma) – Week 6

In the last week prior to midterm, my focus is on the completion of the wrappers(introducing more features like shell type interface), improvement on the stability of remote cuckoo server, and integration with the whole ovizart-ng architecture.

1. completion of the wrappers
In the week 5, I implemented a CLI for wrappers (which users are able to use independently). And during the week 6, I added the shell type interface which allows users to interact with the wrappers. Below is the workflow:

1). Type “python cli.py” in the terminal
2). The help information will be displayed on the screen. And users may follow the instruction to input an url or a file after specifying the analyzer(cuckoo, jsunpackn, or virustotal) .
3). The results would be shown. In some cases, like virustotal and cuckoo sandbox, users may not see the final results instantly because they need to wait in a queue on the server, but get a permlink refers to the results when available instead.
4). Input another url or file. Or ctrl-c to break.

2. improvement on the stability of remote cuckoo server
Before, I used the cuckoo’s default api.py to provide HTTP API access to public. This approach is easy to handle but not stable. In this week, I re-deployed the cuckoo HTTP API on the Apache2. The method is below:
1) Install the apache2
2) Install the mod-wsgi (to get python support)
3) Edit the apache2.conf (adding the site to configuration)
4) Edit the /var/www/yourapp/app.wsgi (basically follow http://bottlepy.org/docs/dev/deployment.html)
5) Restart stuffs, then everything will be ready.

3. integration with the whole ovizart-ng architecture
1) Add __init__.py to every analyzer’s folder which will become a module.
2) Add BaseAnalyzer as the base class for every analyzer(cuckoo, virsutotal, and jsunpackn)
3) Add decorator like @Analyzer(tags=BINARY) to every analyzer.
Special thanks to Gurcan who helped me complete the above things at where I omitted.

BTW, I also fixed the folder issues that may existed if using external library (for example, jsunpack-n source code library folder) by add the value self.jsunpackn_path to ovizconf.py which contains the general Config. Also “import sys” and “sys.path.append”. Finally in this way, the program can access to the library folder.

My plan for next week:
Complete the mid-term evaluation and discuss with my teammates about our next move.

Also special thanks to my mentor Oguz, who just helped me find where I am not doing very well and guided me to learn that I should just come out with my ideas discussing them with teammates, then show them what problems I’ve met and what I’ve done.

Network Analyzer Project Updates (Hao Ma) – Week 5

In the week 5, I was working on the CLI and other issues that integration with tool wrappers.

In order to finish CLI, I learned the argparse( http://docs.python.org/2/library/argparse.html ), which is a pretty nice and easy-to-use module. It helped me to create a CLI for the analyzer wrappers.
And below is the help information of the CLI:

usage: cli.py [-h] [-v] [-c] [-j] [-u URL] [-b BINARY]

An CLI for analyzer wrappers.

optional arguments:
-h, –help show this help message and exit
-v, –virustotal Using Virustotal to analyze binary or url
-c, –cuckoo Using Cuckoo Sandbox to analyze binary
-j, –jsunpackn Using Jsunpack-n to analyze url
-u URL, –url URL Input an url
-b BINARY, –binary BINARY
Input an binary (path)

Thus, we can access the wrappers with a single interface, a more friendly user experience.

In addition, I also was trying to improve the cuckoo stability issue (quite weird, probably not a issue from my side but honeycloud) and package requirements.txt. I believe these will be finished soon before midterm.

For the last week before mid-term, I am going to make the tools more perfect and prepare the materials for mid-term evaluation, which must be exciting.

Network Analyzer Project Updates (Hao Ma) – Week 4

VirusTotal is an awesome tool that could analyze suspicious files and urls(hosts). It may generate the analysis report in html and json. Like below:

https://www.virustotal.com/en/url/b618f0bfbd90176adfcbc2ee9854140f0739063d87807144e26283b0e44791f7/analysis/1370430938/

which provides results like:
URL Scanner Result

ADMINUSLabs Clean site
AlienVault Clean site
Antiy-AVL Clean site
Avira Clean site
BitDefender Clean site
C-SIRT Clean site
CLEAN MX Clean site
,etc.

Files or binaries are similar.

Before you get the analysis reports, you have to submit the request and wait in a queue for a while (won’t be too long). After your report is ready, you can access to the report by visiting a permanent link associated to your request.

The submit procedure can be done using its API, posting to a url, for example “https://www.virustotal.com/vtapi/v2/file/scan” and “https://www.virustotal.com/vtapi/v2/url/scan”.

I’ve written a wrapper and the unittest for it. Due to python’s poor native support for customizing the HTTP POST request, I adopted a simple code which is also recommended by vt.

My task for next week is “Generate results and export to files and implement the CLI”:
I need to enhance the overall user experiences of the current tools and implement a cool and friendly CLI.
In addition, I will also re-test the wrappers in a integrated environment and prepare for mid-term evaluation.