Author Archives: gurcangercek

Network Analyzer Project Updates – Web UI

In this post I’ll introduce our simple Web UI prototype.

Before we open the browser we need to start 2 different scripts under ovizart-ng/bin/ directory. First one is the daemon service, which is a basically a small REST API providing HTTP  Server. To start;

./api_server.py start

This command will start a https server on localhost:9009 in order to change this values you can use this syntax;

api_server.py [-h] [--host HOST] [--port PORT] [--ssl] {start,stop,restart}

Second command is responsible for  starting Web UI, which is based on Django 1.5. To start;

./ui_server.py

this command will start Web UI on localhost:8000. Now we are ready; open a browser and on address bar write http://localhost:8000/.

This screen will show up for login and daemon settings. Before we move on, Daemon Options will be moved to a configuration file, for the ease of development and debugging I put those fields on the login form. These options should match with daemon parameters, for default parameters user do not need to change anything.

In order to login, a user must be created with create_user.py script under directory ./ovizart-ng/bin/. In our example the user and password is admin. This is not a default user account. Actually, system has no default users, one must be created right after installation.
WebUI-1

After login, (because of the first login) system does not contain any analysis. In order to start one click on the ‘New’ button on the left corner.

WebUI-2This screen needs some makeup, but it has some nice feature. For example besides uploading your pcap file you can upload your analyzers as well. so that you don’t need to have an account on the core machine to use your own analyzer. I’m well aware that this feature could be very dangerous. I’m planing to take 2 measures in order to improve security. First, improving user management by adding roles and rights, so that only certain users will have right to upload analyzers. Second one is sand-boxing. Running analyzer module in a sandbox will make this feature a little bit safer.

WebUI-3Select your pcap file to upload and click on ‘Upload & Start’ button. Your next screen will be this one;WebUI-4

After some time (system does not have a progressbar to show the current status of the evaluation), click on the ‘Browse’ button or refresh the page to see changed status of the analysis. If you want to delete an analysis click on the checkbox on the left side of the analysis and click on the ‘Delete’ button. This action can not be done, will delete each information, files, reports, etc. generated during that analysis.

WebUI-5Finished analyses have a summary on the rightmost column, number of packet, name of the pcap file and number of streams extracted from given pcap file. Clicking on ID will open the details screen.

WebUI-8At the top part we have the summary section which contains basic information about the given pcap file. The next section contains the information about the streams extracted from given pcap file. Stream list is a collapsible table. Each row of this table starts Application/Transport Layer protocol information. Then we have standard stream identifiers Source IP, Source Port, Destination IP, Destination Port. Number of Packets follows the indetifiers.

On rightmost column we can observe an icon of file and magnifier. File means that system extracted some file(s) from that specific stream. Magnifier means that system has analyzer reports for that specific stream’s extracted files.

WebUI-6 Clicking on a row will expand that row and show additional info about that stream.

  • Pcap file: clicking on the filename will start download of that stream specific pcap file.
  • Reassembled Traffic: Those links provide reconstructed application layer traffic in a file for further analysis/study/examination. You can see 3 different links. They are
    A -> B, this file contains all requests made by A.
    A <- B, this file contains all responses given by B
    A <-> B, this file contains whole request response pair between A and B.
    clicking on the links will start the download of those files.
  • Attachments: This section contains information about extracted files from that stream. On the right column you can see the mime-type of the extracted file as well. Clicking on the link will start the download of extracted file.
  • Analyzer Reports: Current system does have Virus Total and Cuckoo wrappers as analyzers. Clicking on those links will open a new tab for the results to see. Because of the limitations analyze results may take some time to be ready. Here is a sample screen-shot from virustotal.

WebUI-7This is our first prototype to show the infrastructure in a more user-friendly way :)

Cheers,
Gurcan

Network Analyzer Project Updates – Week 12

3 scripts added under ovizart-ng/bin/ directory to control ovizart system.

  • create user.py: A basic tool to create ovizart users to access the system.
    $ ./create_user.py 
    usage: create_user.py [-h]
                          <username> <password> <name> <surname>
                          <email@example.com>
    create_user.py: error: too few arguments
    
    $ ./create_user.py ggercek 123456 Gurcan GERCEK gurcangercek@gmail.com
    User created successfully.
    Now retrieving user for testing
    User retrieved successfully, user_id: 5239e1641e75ed0e0845d715
  • api_server.py: A basic Daemon management script, which trigger the Core & REST API system start/stop/restart
    $ ./api_server.py 
    usage: api_server daemon script [-h] [--host HOST] [--port PORT] [--ssl]
                                    {start,stop,restart}
    api_server daemon script: error: too few arguments
    
    $ ./api_server.py --host localhost --port 9009 --ssl start
    value: 9009
    $ ./api_server.py stop
    (No output if successfully terminated)

    –host HOST: specifies the binding address of the server. (default: localhost)
    –port PORT: specifies the binding port. (default: 9009)
    –ssl: specifies whether the server will run over http or https. (default: false)

  • ui_server.py.py: A basic start script for WEB UI. Use CTRL+C to stop execution.
    $ ./ui_server.py 
    Validating models...
    
    0 errors found
    September 18, 2013 - 14:24:09
    Django version 1.5.1, using settings 'web.settings'
    Development server is running at http://127.0.0.1:8000/
    Quit the server with CONTROL-C.

For the Web UI following items added.

  • Remove analysis option added: This feature will clean up everything ( reassembled traffic, extracted files, splitted pcap files, etc) related with that analysis
  • Listing details of streams
  • Dynamic Analyzer Loading: While uploading pcap files, users can upload their custom analyzers as well, so that it will be easier to extend the tool. But be cautious and use this feature at your own risk, I will plan to add sandboxing for this feature but that will take some time.

I will post the details of Web UI with screenshots.

Cheers,
Gurcan

Network Analyzer Project Updates – Week 11

This week I was working on Web UI. A simple UI implementation based on Django 1.5. Actually it should be a basic task but using OvizartProxy for the whole backend of the system makes it a little bit harder. Here is why;

Login/Logout feature. Django provides a nice user authentication/authorization system but in our case we can not mainly depend on it because we already implemented a small user management system for our core module. In order to use core module, each user must have a valid username/password pair. First problem was the integrating the authentication system of ovizart with Django. Luckily Django has a great feature as ‘Authentication Backend’, which will allow integration of external authentication systems. Here is a good starting point; https://docs.djangoproject.com/en/dev/topics/auth/default/

This was a nice opportunity to learn more about django. You can find the backend implementation here: https://github.com/honeynet/ovizart-ng/blob/ovizart-ng-devel/web/web/backend.py

Another trick to apply is I tried to extend the Django User class but this was required more time because it requires more configuration. So I used a work around, which was not elegant but saved the day for now. For each logged in user we need to have a OvizartProxy instance associated with them. The easiest way to do is assign OvizartProxy instances to a member variable of User objects, which can be done __dict__; eg.

op = OvizartProxy(protocol, host, port)
...
user.__dict__['ovizart'] = op
user.__dict__['userid'] = userid

with this approach I’m able to reach ovizartProxy instances in views with the following code;

def upload_file(request):
    ...
    op = request.user.ovizart

We can deploy our Web UI on any host that can communicate with an ovizart core daemon, to reduce the work load or restricted access.

Cheers,
Gurcan

Network Analyzer Project Updates – Week 10

REST API Structure & OvizartProxy

Current REST API functionality is as follows:

Base URL: http://localhost:9009/

Url Method / Content-TYPE Description
/login
{username:, password:}
POST
application/json
Authenticate user and generates a cookie. All of the API calls require that cookie value.
/upload/<filename>
<file-content>
POST
application/octet-stream or
multipart/form-data
Uploads the specified file to core daemon, in case of name conflict system will rename the given file as <base_name>_#.<extension> and returns the given name to client.. eg, smtp.pcap -> smtp_1.pcap
File must be a pcap file otherwise users cookie will be be terminate.
/start POST
application/json
After uploading pcap file, this will trigger the core system to start the analyze. This is an async call. This call will return id of analysis started.
/analysis/ GET
application/json
This call will return a summary analysis’ crated by current user.
/analysis/<analysisId>  GET
application/json
This call will return details of given analysis
/analysis/<analysisId>  DELETE
application/json
This call will delete all the information related analysis. All records from database, all the attachment files, both the original and separated pcap files.
/analyzer PUT
application/json
This call will allow user to load custom analyzer into the system. This feature is in a very early stage and does not have security checks. It could be dangerous.

Although REST API can be used by any kind of client, we developed a counterpart of REST API which we call OvizartProxy. This class provides an abstraction interface to REST API calls so that we can use this class for remote connections as well as local connetions. The usage is quite straight forward here is an example usage of OvizartProxy with ipython;


In [1]: from core.ovizart_proxy import OvizartProxy

In [2]: op = OvizartProxy('http', 'localhost', '9009')

In [3]: op = OvizartProxy('https', 'localhost', '9009')

In [4]: op.login('ggercek', '123456')
Out[4]: 
{u'Status': u'OK',
 u'userid': u'5227e32268caca1a3c047374',
 u'username': u'ggercek'}

In [5]: op.uploadFile('./test/pcap/test-http.pcap')
Out[5]: {u'Filesize': 326754, u'Status': u'OK', u'filename': u'test-http.pcap'}

In [6]: op.start()
Out[6]: {u'AnalysisId': u'522a734968caca1103775079', u'Status': u'INIT'}

In [7]: op.getAnalysis()
Out[7]: 
[{u'_id': u'5227f5b968caca205d6fba93',
  u'files': [{u'filename': u'/home/hforge/src/ovizart-ng/upload/smtp_2.pcap',
    u'numberOfPacket': 60,
    u'numberOfStreams': 4}],
  u'startTime': 1378350521.216652,
  u'status': u'FINISHED'},
 {u'_id': u'5227f5d468caca205d6fba94',
  u'files': [],
  u'startTime': 1378350548.074157,
  u'status': u'INIT'},
 {u'_id': u'522a734968caca1103775079',
  u'files': [{u'filename': u'/home/hforge/src/ovizart-ng/upload/test-http.pcap',
    u'numberOfPacket': 483,
    u'numberOfStreams': 21}],
  u'startTime': 1378513737.705265,
  u'status': u'FINISHED'}]

In [8]: op.getAnalysis('522a734968caca1103775079')
Out[8]: 
{u'_id': u'522a734968caca1103775079',
 u'config': {u'cuckoo_html_enabled': True,
  u'cuckoo_ip': u'81.167.148.242',
  u'cuckoo_port': 8090,
  u'cuckoo_tcpdump_enabled': True,
  u'cuckoo_timeout': 60,
  u'exclude_analyzer': [],
  u'input_files': [u'/home/hforge/src/ovizart-ng/upload/test-http.pcap'],
  u'is_cuckoo_available': True,
  u'jsunpackn_path': u'',
  u'output_folder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/',
  u'vt_apikey': u'545b985a35ca91e6bd5232c9cfb3549dd5e74a506ea960336472f86be156ec8d'},
 u'data': [{u'_Data__data': {u'stream': {u'dstIp': u'10.1.1.1',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3188_10.1.1.1_80',
     u'last_ts': 1100903355.562335,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3188_10.1.1.1_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3188_10.1.1.1_80/6_10.1.1.101_3188_10.1.1.1_80.pcap',
     u'pktCount': 14,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3188',
     u'startTime': u'1100903355.43'}},
   u'_Data__tags': {u'PLAIN_TEXT': [u'_Websidan_index.html'],
    u'app_layer_protocol': u'HTTP',
    u'attachments': [[u'_Websidan_index.html',
      u'text/html; charset=iso-8859-1',
      u'PLAIN_TEXT']],
    u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'10.1.1.1',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3199_10.1.1.1_80',
     u'last_ts': 1100903361.063142,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3199_10.1.1.1_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3199_10.1.1.1_80/6_10.1.1.101_3199_10.1.1.1_80.pcap',
     u'pktCount': 20,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3199',
     u'startTime': u'1100903360.9'}},
   u'_Data__tags': {u'ANALYZER_RESPONSES': [[u'CK_RESPONSE',
      {u'task_id': 192, u'url': u'http://81.167.148.242:8090/tasks/view/192'}],
     [u'CK_RESPONSE',
      {u'task_id': 193, u'url': u'http://81.167.148.242:8090/tasks/view/193'}],
     [u'VT_RESPONSE',
      u'{"scan_id": "9c5672ca9f1e8518ccb5e336efdcd37c8612a7e7ff0abf29aba4707322c10b41-1378513741", "sha1": "4c7e8bba5bb5744a7cc735cdcd4aeb8633269980", "resource": "9c5672ca9f1e8518ccb5e336efdcd37c8612a7e7ff0abf29aba4707322c10b41", "response_code": 1, "sha256": "9c5672ca9f1e8518ccb5e336efdcd37c8612a7e7ff0abf29aba4707322c10b41", "permalink": "https://www.virustotal.com/file/9c5672ca9f1e8518ccb5e336efdcd37c8612a7e7ff0abf29aba4707322c10b41/analysis/1378513741/", "md5": "0951a03339a81693ed8987c43b6dd1ba", "verbose_msg": "Scan request successfully queued, come back later for the report"}']],
    u'BINARY': [u'_Websidan_2004-07-SeaWorld_320_DSC07859.JPG'],
    u'app_layer_protocol': u'HTTP',
    u'attachments': [[u'_Websidan_2004-07-SeaWorld_320_DSC07859.JPG',
      u'image/jpeg; charset=binary',
      u'BINARY']],
    u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'209.225.0.6',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3192_209.225.0.6_80',
     u'last_ts': 1100903358.11618,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3192_209.225.0.6_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3192_209.225.0.6_80/6_10.1.1.101_3192_209.225.0.6_80.pcap',
     u'pktCount': 14,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3192',
     u'startTime': u'1100903356.35'}},
   u'_Data__tags': {u'app_layer_protocol': u'HTTP', u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'10.1.1.1',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3177_10.1.1.1_80',
     u'last_ts': 1100903354.296049,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3177_10.1.1.1_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3177_10.1.1.1_80/6_10.1.1.101_3177_10.1.1.1_80.pcap',
     u'pktCount': 10,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3177',
     u'startTime': u'1100903354.16'}},
   u'_Data__tags': {u'PLAIN_TEXT': [u'_'],
    u'app_layer_protocol': u'HTTP',
    u'attachments': [[u'_', u'text/html; charset=us-ascii', u'PLAIN_TEXT']],
    u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'209.225.0.6',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3193_209.225.0.6_80',
     u'last_ts': 1100903358.656305,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3193_209.225.0.6_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3193_209.225.0.6_80/6_10.1.1.101_3193_209.225.0.6_80.pcap',
     u'pktCount': 15,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3193',
     u'startTime': u'1100903356.74'}},
   u'_Data__tags': {u'app_layer_protocol': u'HTTP', u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'10.1.1.1',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3200_10.1.1.1_80',
     u'last_ts': 1100903365.542586,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3200_10.1.1.1_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3200_10.1.1.1_80/6_10.1.1.101_3200_10.1.1.1_80.pcap',
     u'pktCount': 209,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3200',
     u'startTime': u'1100903364.99'}},
   u'_Data__tags': {u'ANALYZER_RESPONSES': [[u'CK_RESPONSE',
      {u'task_id': 194, u'url': u'http://81.167.148.242:8090/tasks/view/194'}],
     [u'CK_RESPONSE',
      {u'task_id': 195, u'url': u'http://81.167.148.242:8090/tasks/view/195'}],
     [u'VT_RESPONSE',
      u'{"scan_id": "2e79767d8e87877225e7bc798b93001e0d90f6ab1c62238d55efd77e6daaca54-1378513744", "sha1": "efcca7c4ca7da6cfb45a1ecf28ba884ec76b9355", "resource": "2e79767d8e87877225e7bc798b93001e0d90f6ab1c62238d55efd77e6daaca54", "response_code": 1, "sha256": "2e79767d8e87877225e7bc798b93001e0d90f6ab1c62238d55efd77e6daaca54", "permalink": "https://www.virustotal.com/file/2e79767d8e87877225e7bc798b93001e0d90f6ab1c62238d55efd77e6daaca54/analysis/1378513744/", "md5": "74da406ca3055d0e56080b796c670ee3", "verbose_msg": "Scan request successfully queued, come back later for the report"}']],
    u'BINARY': [u'_Websidan_2004-07-SeaWorld_fullsize_DSC07858.JPG'],
    u'app_layer_protocol': u'HTTP',
    u'attachments': [[u'_Websidan_2004-07-SeaWorld_fullsize_DSC07858.JPG',
      u'image/jpeg; charset=binary',
      u'BINARY']],
    u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'209.225.0.6',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3187_209.225.0.6_80',
     u'last_ts': 1100903356.992361,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3187_209.225.0.6_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3187_209.225.0.6_80/6_10.1.1.101_3187_209.225.0.6_80.pcap',
     u'pktCount': 13,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3187',
     u'startTime': u'1100903355.42'}},
   u'_Data__tags': {u'app_layer_protocol': u'HTTP', u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'209.225.0.6',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3191_209.225.0.6_80',
     u'last_ts': 1100903358.114687,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3191_209.225.0.6_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3191_209.225.0.6_80/6_10.1.1.101_3191_209.225.0.6_80.pcap',
     u'pktCount': 14,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3191',
     u'startTime': u'1100903356.16'}},
   u'_Data__tags': {u'app_layer_protocol': u'HTTP', u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'209.225.0.6',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3185_209.225.0.6_80',
     u'last_ts': 1100903357.433873,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3185_209.225.0.6_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3185_209.225.0.6_80/6_10.1.1.101_3185_209.225.0.6_80.pcap',
     u'pktCount': 13,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3185',
     u'startTime': u'1100903355.39'}},
   u'_Data__tags': {u'app_layer_protocol': u'HTTP', u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'10.1.1.101',
     u'dstPort': u'None',
     u'fileHandler': None,
     u'key': u'6_209.225.11.237_10.1.1.101',
     u'last_ts': 1100903355.113119,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_209.225.11.237_10.1.1.101',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_209.225.11.237_10.1.1.101/6_209.225.11.237_10.1.1.101.pcap',
     u'pktCount': 1,
     u'protocol': 6,
     u'srcIP': u'209.225.11.237',
     u'srcPort': u'None',
     u'startTime': u'1100903355.11'}},
   u'_Data__tags': {u'app_layer_protocol': u'UNKNOWN',
    u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'209.225.0.6',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3183_209.225.0.6_80',
     u'last_ts': 1100903356.737359,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3183_209.225.0.6_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3183_209.225.0.6_80/6_10.1.1.101_3183_209.225.0.6_80.pcap',
     u'pktCount': 13,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3183',
     u'startTime': u'1100903355.36'}},
   u'_Data__tags': {u'app_layer_protocol': u'HTTP', u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'209.225.0.6',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3184_209.225.0.6_80',
     u'last_ts': 1100903356.741105,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3184_209.225.0.6_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3184_209.225.0.6_80/6_10.1.1.101_3184_209.225.0.6_80.pcap',
     u'pktCount': 13,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3184',
     u'startTime': u'1100903355.36'}},
   u'_Data__tags': {u'app_layer_protocol': u'HTTP', u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'10.1.1.101',
     u'dstPort': u'None',
     u'fileHandler': None,
     u'key': u'6_209.225.0.6_10.1.1.101',
     u'last_ts': 1100903358.380375,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_209.225.0.6_10.1.1.101',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_209.225.0.6_10.1.1.101/6_209.225.0.6_10.1.1.101.pcap',
     u'pktCount': 18,
     u'protocol': 6,
     u'srcIP': u'209.225.0.6',
     u'srcPort': u'None',
     u'startTime': u'1100903356.13'}},
   u'_Data__tags': {u'app_layer_protocol': u'UNKNOWN',
    u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'10.1.1.1',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3195_10.1.1.1_80',
     u'last_ts': 1100903357.633431,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3195_10.1.1.1_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3195_10.1.1.1_80/6_10.1.1.101_3195_10.1.1.1_80.pcap',
     u'pktCount': 10,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3195',
     u'startTime': u'1100903357.41'}},
   u'_Data__tags': {u'PLAIN_TEXT': [u'_Websidan_dagbok_dagbok.html'],
    u'app_layer_protocol': u'HTTP',
    u'attachments': [[u'_Websidan_dagbok_dagbok.html',
      u'text/html; charset=us-ascii',
      u'PLAIN_TEXT']],
    u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'10.1.1.1',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3190_10.1.1.1_80',
     u'last_ts': 1100903355.870696,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3190_10.1.1.1_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3190_10.1.1.1_80/6_10.1.1.101_3190_10.1.1.1_80.pcap',
     u'pktCount': 19,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3190',
     u'startTime': u'1100903355.54'}},
   u'_Data__tags': {u'ANALYZER_RESPONSES': [[u'CK_RESPONSE',
      {u'task_id': 196, u'url': u'http://81.167.148.242:8090/tasks/view/196'}],
     [u'CK_RESPONSE',
      {u'task_id': 197, u'url': u'http://81.167.148.242:8090/tasks/view/197'}],
     [u'VT_RESPONSE',
      u'{"scan_id": "8acd70621921083a1ab4394ed7dc9d17844d094e3b4cede1039cb6afdceb6570-1378513746", "sha1": "ebaf3b1b6a09fe3f9d2a2e2e6bca7d1491ded813", "resource": "8acd70621921083a1ab4394ed7dc9d17844d094e3b4cede1039cb6afdceb6570", "response_code": 1, "sha256": "8acd70621921083a1ab4394ed7dc9d17844d094e3b4cede1039cb6afdceb6570", "permalink": "https://www.virustotal.com/file/8acd70621921083a1ab4394ed7dc9d17844d094e3b4cede1039cb6afdceb6570/analysis/1378513746/", "md5": "f5d0c27ca554a8564e0ae1edd3ea002b", "verbose_msg": "Scan request successfully queued, come back later for the report"}']],
    u'BINARY': [u'_Websidan_images_sydney.jpg'],
    u'app_layer_protocol': u'HTTP',
    u'attachments': [[u'_Websidan_images_sydney.jpg',
      u'image/jpeg; charset=binary',
      u'BINARY']],
    u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'209.225.11.237',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3179_209.225.11.237_80',
     u'last_ts': 1100903355.609257,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3179_209.225.11.237_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3179_209.225.11.237_80/6_10.1.1.101_3179_209.225.11.237_80.pcap',
     u'pktCount': 13,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3179',
     u'startTime': u'1100903354.28'}},
   u'_Data__tags': {u'app_layer_protocol': u'HTTP', u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'10.1.1.1',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3197_10.1.1.1_80',
     u'last_ts': 1100903360.924474,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3197_10.1.1.1_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3197_10.1.1.1_80/6_10.1.1.101_3197_10.1.1.1_80.pcap',
     u'pktCount': 12,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3197',
     u'startTime': u'1100903360.81'}},
   u'_Data__tags': {u'PLAIN_TEXT': [u'_Websidan_dagbok_2004_28_dagbok.html'],
    u'app_layer_protocol': u'HTTP',
    u'attachments': [[u'_Websidan_dagbok_2004_28_dagbok.html',
      u'text/html; charset=iso-8859-1',
      u'PLAIN_TEXT']],
    u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'10.1.1.1',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3198_10.1.1.1_80',
     u'last_ts': 1100903361.044274,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3198_10.1.1.1_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3198_10.1.1.1_80/6_10.1.1.101_3198_10.1.1.1_80.pcap',
     u'pktCount': 19,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3198',
     u'startTime': u'1100903360.9'}},
   u'_Data__tags': {u'ANALYZER_RESPONSES': [[u'CK_RESPONSE',
      {u'task_id': 198, u'url': u'http://81.167.148.242:8090/tasks/view/198'}],
     [u'CK_RESPONSE',
      {u'task_id': 199, u'url': u'http://81.167.148.242:8090/tasks/view/199'}],
     [u'VT_RESPONSE',
      u'{"scan_id": "ff9140064b9b70609962b4430ce3090be373e8f7e876e86497f01e9907b82247-1378513748", "sha1": "32e02bbe5a1a6ec90af3f75b59af5942ae83949a", "resource": "ff9140064b9b70609962b4430ce3090be373e8f7e876e86497f01e9907b82247", "response_code": 1, "sha256": "ff9140064b9b70609962b4430ce3090be373e8f7e876e86497f01e9907b82247", "permalink": "https://www.virustotal.com/file/ff9140064b9b70609962b4430ce3090be373e8f7e876e86497f01e9907b82247/analysis/1378513748/", "md5": "835d8e7a12d31c4bbe4eeff7b4b5ab3b", "verbose_msg": "Scan request successfully queued, come back later for the report"}']],
    u'BINARY': [u'_Websidan_2004-07-SeaWorld_320_DSC07858.JPG'],
    u'app_layer_protocol': u'HTTP',
    u'attachments': [[u'_Websidan_2004-07-SeaWorld_320_DSC07858.JPG',
      u'image/jpeg; charset=binary',
      u'BINARY']],
    u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'209.225.0.6',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3194_209.225.0.6_80',
     u'last_ts': 1100903358.606544,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3194_209.225.0.6_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3194_209.225.0.6_80/6_10.1.1.101_3194_209.225.0.6_80.pcap',
     u'pktCount': 14,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3194',
     u'startTime': u'1100903356.96'}},
   u'_Data__tags': {u'app_layer_protocol': u'HTTP', u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'10.1.1.1',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3189_10.1.1.1_80',
     u'last_ts': 1100903355.703068,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3189_10.1.1.1_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3189_10.1.1.1_80/6_10.1.1.101_3189_10.1.1.1_80.pcap',
     u'pktCount': 17,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3189',
     u'startTime': u'1100903355.54'}},
   u'_Data__tags': {u'ANALYZER_RESPONSES': [[u'CK_RESPONSE',
      {u'task_id': 200, u'url': u'http://81.167.148.242:8090/tasks/view/200'}],
     [u'CK_RESPONSE',
      {u'task_id': 201, u'url': u'http://81.167.148.242:8090/tasks/view/201'}],
     [u'VT_RESPONSE',
      u'{"scan_id": "bcdc6e9ee31daa151e643978d95c41959c01a9ef223fe362f46b4c52e464ee23-1378513749", "sha1": "96df67be79c09020bb7967ead3fa3a062eb2f7dc", "resource": "bcdc6e9ee31daa151e643978d95c41959c01a9ef223fe362f46b4c52e464ee23", "response_code": 1, "sha256": "bcdc6e9ee31daa151e643978d95c41959c01a9ef223fe362f46b4c52e464ee23", "permalink": "https://www.virustotal.com/file/bcdc6e9ee31daa151e643978d95c41959c01a9ef223fe362f46b4c52e464ee23/analysis/1378513749/", "md5": "ba1a813191165661b6cc5ef4344141c2", "verbose_msg": "Scan request successfully queued, come back later for the report"}']],
    u'BINARY': [u'_Websidan_images_bg2.jpg'],
    u'app_layer_protocol': u'HTTP',
    u'attachments': [[u'_Websidan_images_bg2.jpg',
      u'image/jpeg; charset=binary',
      u'BINARY']],
    u'data_source': u'PCAP'}},
  {u'_Data__data': {u'stream': {u'dstIp': u'10.1.1.1',
     u'dstPort': u'80',
     u'fileHandler': None,
     u'key': u'6_10.1.1.101_3196_10.1.1.1_80',
     u'last_ts': 1100903359.24759,
     u'outputFolder': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3196_10.1.1.1_80',
     u'pcapFile': None,
     u'pcapFileName': u'/home/hforge/src/ovizart-ng/test/output/analysis_20130907_032857_706265/test-http.pcap/6_10.1.1.101_3196_10.1.1.1_80/6_10.1.1.101_3196_10.1.1.1_80.pcap',
     u'pktCount': 12,
     u'protocol': 6,
     u'srcIP': u'10.1.1.101',
     u'srcPort': u'3196',
     u'startTime': u'1100903359.07'}},
   u'_Data__tags': {u'PLAIN_TEXT': [u'_Websidan_dagbok_2004_dagbok.html'],
    u'app_layer_protocol': u'HTTP',
    u'attachments': [[u'_Websidan_dagbok_2004_dagbok.html',
      u'text/html; charset=us-ascii',
      u'PLAIN_TEXT']],
    u'data_source': u'PCAP'}}],
 u'files': [{u'filename': u'/home/hforge/src/ovizart-ng/upload/test-http.pcap',
   u'numberOfPacket': 483,
   u'numberOfStreams': 21}],
 u'startTime': 1378513737.705265,
 u'status': u'FINISHED',
 u'user': u'5227e32268caca1a3c047374'}

Cheers,
Gurcan

Network Analyzer Project Updates – Week 9

This week I was working on Mongo DB Integration and Built-in Web Server improvements.

Mongo DB was quite easy to start with, install it with apt-get install mongodb then write mongod to start playing. A nice tutorial can be found from here: http://docs.mongodb.org/manual/tutorial/getting-started/

DB system of the project is quite simple. Currently we are storing all the data in 2 collections, which we will split up in future. Analysis and Users, all the data generated by the system is stored as json in our object models so it is too easy to map those data to mongodb. Just one relation exists in our system, which is obvious :) User – Analysis. Currently a user can only check it’s own analysis results but I think it would be a nice feature to create a collaborative environment for such tool. So that users can share their analyze results or even their data fully or partially. By means of partially we are splitting all the streams we detect, so rather than sharing all data, users may choose to share some part of it to anounce some interesting results or to get feedback or to take advise where he/she got stuck.

As I said I worked on builtin http server, but it was not fun as it supposed to be. Choosing BaseHttpServer for REST API usage was a bad idea. I had to implement SSL/TSL support and session management also file upload capabilities(form based and stream based), which took some time. But now it seems everything’s fine.

Also authentication support added to API decorator, which is integrated with web server. Not a big change but useful for me :) An example of old API:

@API(method='GET', url=r"^/(?P<username>\w+)/(?P<password>\w+)/?$")
def login(data):
    ...

New API version:

@API(method='GET', url=r"^/(?P<username>\w+)/(?P<password>\w+)/?$", isAuth=False)
def login(data):
    ...

Default value for isAuth is True and this parameters forces web server to check its authentication status over clients cookie/IP Address pair and calls the decorated function if user has been authenticated.

Implmentation of built-in server is not very fancy but feel free to comment/criticize or share any idea to improve it.

Cheers,
Gurcan

Network Analyzer Project Updates – Week 8

Reassembly Module

After extracting streams from pcap files we need to examine the traffic. Although this could be done by inspecting each packets which is what we are doing it on Tagger module for protocol detection reasons. But this approach is easy to bypass using fragmentation. Another reason is to inspect the whole traffic in a higher level. This module is composed of two layers; BaseReassembler and ProtocolReassembler.

First layer, BaseReassembler is responsible for running and interacting with justniffer. It will save the traffic into 3 files for further analysis. The files contains the reconstructed traffic between two hosts(A, B), with names of ‘request.traffic'(A->B), ‘response.traffic'(A<-B) and ‘total.traffic'(AB). Here is the sample content of files from a http connection.

A -> B, request.traffic:

GET / HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) Opera 7.11  [en]
Host: 10.1.1.1
Accept: application/x-shockwave-flash,text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,text/css,*/*;q=0.1
Accept-Language: en
Accept-Charset: windows-1252, utf-8, utf-16, iso-8859-1;q=0.6, *;q=0.1
Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0
Connection: Keep-Alive

A <- B, response.traffic:

HTTP/1.1 200 OK
Date: Sat, 20 Nov 2004 10:21:06 GMT
Server: Apache/2.0.40 (Red Hat Linux)
Last-Modified: Mon, 08 Mar 2004 20:27:54 GMT
ETag: "46eed-a0-800ce680"
Accept-Ranges: bytes
Content-Length: 160
Connection: close
Content-Type: text/html; charset=ISO-8859-1

<html>
<head>
<title>
Ronnie sahlbergs Websida
</title>
</head>
<body>
<a href="./Websidan/index.html">Familjen Sahlbergs Websida</a>
</body>
</html>

A <-> B, total.traffic:

 GET / HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) Opera 7.11  [en]
Host: 10.1.1.1
Accept: application/x-shockwave-flash,text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,text/css,*/*;q=0.1
Accept-Language: en
Accept-Charset: windows-1252, utf-8, utf-16, iso-8859-1;q=0.6, *;q=0.1
Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Sat, 20 Nov 2004 10:21:06 GMT
Server: Apache/2.0.40 (Red Hat Linux)
Last-Modified: Mon, 08 Mar 2004 20:27:54 GMT
ETag: "46eed-a0-800ce680"
Accept-Ranges: bytes
Content-Length: 160
Connection: close
Content-Type: text/html; charset=ISO-8859-1

<html>
<head>
<title>
Ronnie sahlbergs Websida
</title>
</head>
<body>
<a href="./Websidan/index.html">Familjen Sahlbergs Websida</a>
</body>
</html>

The second layer is ProtocolReassembler. This layer is responsible for extracting application layer info or useful data such as transferred files, email body, etc. In our system there are two kinds of ProtocolReassembler exists, HTTP and SMTP and our tool is able to extract files in HTTP and SMTP streams such as html, js, etc. and save under the attachments folder of related stream.

Adding new ProtocolReassembler is quite simple. Here is an skeleton of ProtocolReassembler;

class MySuperProtocolReassembler(BaseReassembler):
    target = 'MySuperProtocol'

    def __init__(self, outputFolder):
        BaseReassembler.__init__(self, outputFolder)
        # Init some variable here, if you need

    def processRequest(self, request):
        # Make sure to call this line
        BaseReassembler.processRequest(self, request)
        # process request here !!!

    def processResponse(self, response):
        # Make sure to call this line
        BaseReassembler.processResponse(self, response)
        # process response here !!!

It is important to extend BaseReassembler and define ‘target’ class attribute. Any stream object contains the ‘MySuperProtocol‘ tag will be processed by this class’ instance. You can find full code for http reassembler here: https://github.com/honeynet/ovizart-ng/blob/ovizart-ng-devel/reassembler/http_reassembler.py

Last step for adding, we need to register our ProtocolReassembler to reassembler/__init__.py by updating ‘parsers’ dictionary. I’ll add decorators for this structure later for simplicity. Here is the sample;

parsers = {
    BaseReassembler.target: 'BaseReassembler',
    SMTPReassembler.target: 'SMTPReassembler',
    HTTPReassembler.target: 'HTTPReassembler',
    # Register MySuperProtocolReassembler
    MySuperProtocolReassembler.target, 'MySuperProtocolReassembler'
}

That’s all about reassembler module.

Cheers,
Gurcan

Network Analyzer Project Updates – Week 7

This week I implemented http reassembler module, so that we can extract files from http sessions.Most of time I was working on the cli tool to improve prototype a bit. In final version (on branch ovizart-ng-devel) ovizcli.py takes pcap file(s) and output folder. Then print the results in json. Not very fancy but, soon we will print it in more readable way.

ovizcli.py -i some.pcap -o /tmp/output/

We also provide direct access to our wrappers(cuckoo, virustotal and jsunpack-n) over ovizcli.py

An example usage;
To run cuckoo wrapper on a exe file

ovizcli.py -i some.exe -ck

To use virustotal with some.exe and http://someURL/

ovizcli.py -i some.exe http://someURL/ -vt

To use jsunpack-n wrapper

ovizcli.py -i some.js -js

You can combine the options and inputs, tool will detect input types and group them and  wrappers will process everything they can. For example -vt will process binary and url inputs, -ck will process binary inputs, -js will process javascript inputs and given pcap files are processed by ovizart it self.

ovizcli.py -i some.pcap some.exe some.js http://someURL/ http://anotherURL/ -js -vt -ck -o /tmp/output/

That’s all for now.

Network Analyzer Project Updates – Week 6

This week is totaly taken by integration and preparations for midterm evaluation. Until this week all the work is done tested by simple unittests but I didn’t test the full integration of modules, which was a very bad idea :)

All modules are now working fine, Simple work flow is as follows;

  1. Given pcap files splitted into pcap files based on TCP, UDP or IP streams.
  2. Found streams processed in tagger modules: currently just detecting application layer protocols
  3. Reassembly the traffic between host A, B and save to 3 files: request.traffic(A -> B), response.traffic(A <- B) and total.traffic(A <-> B). We can extend reassemblers for protocol based needs. For example we have SMTPReassembler class which extends BaseReassembler, which is responsible for step 3. SMTPReassembler is responsible for finding and extracting attachments in given SMTP stream. Currently we have only SMTP reassembler but HTTP will be added soon.
  4. Run the analyzers on streams, Currently tool has 3 analyzers, which are implemented by HAO: http://gsoc2013.honeynet.org/author/haoma/. These analyzers take the files extracted and send it to external analyzers and save the reports. Those are Cuckoo for Binary files, jsunpack-n for javascript files and Virus Total for Binary files as well.
  5. Reporter module organizes the data collected and print out the results.(Currently a work in progress)

I will write an entry to demonstrate a sample analysis of SMTP traffic with an attachment(malware)

Network Analyzer Project Updates – Week 5

This week I made some refactoring. A sample is decorators, they not only check the method name of applied class but also checks for given arguments name as well as argument number. I thought this will provide an early warning feature about whether implemented module classes, such as taggers, analyzers will work with the system without any problem or not.

Besides refactoring I implemented tagger module. This module is responsible for tagging the given streams based on the collected/calculated data, such as whether the stream is encrypted or not or what is the application layer protocol of given stream, etc.

As first tagger, I implemented application layer protocol tagger. Currently our tool can detect 3 protocols, SMTP, HTTP and FTP, based on packet-based signatures. I used Scapy for this job. Scapy is built upon layer logic each packet composed of layers and the layers can be binded with each other as 2 ways. You can use bind_layers function or guess_payload_class.

bind_layers(IP, TCP, frag=0, proto=6)

In this example IP layer is binded to TCP layer based on IP layer’s fields’ values of frag=0 and proto=6. Any packet contains IP layer and satisfies given condition will be processed as a TCP packet. You can split binded layers with split_layers function.

split_layers(IP, TCP)

Until TCP layer this approach is OK but what about the application layer protocols? We can not use fields of TCP layer in or to decide what is located in payload. So we will use second method guess_payload_class. This method the payload to check it’s content and decide what could be the class of it. A sample;

def guess_payload_class(self, payload):
    if re.match(r'^(EHLO|DATA|AUTH|MAIL|RCPT|QUIT).*', payload, re.IGNORECASE):
        return SMTPRequest

SMTPRequest should be a subclass of scapy.packet.Packet class. For further information please check Scapy documentation: http://trac.secdev.org/scapy/wiki/BuildAndDissect#Bindinglayers

SMTPRequest class does not have to parse whole packet fields to make it work. But later we can add protocol based state analysis as well so that I used some parts of dissectors project: https://github.com/cssaheel/dissectors, which was a GSoC 2012 project for improving Cuckoo Sandbox. You can check the details here: https://honeynet.org/gsoc2012/slot3

How I implemented protocol detection mechanism is quite simple. First of all I had to override of guess_payload_class functions of both TCP and UDP layers, which was easy(from now on TCP and UDP classes in Scapy will be mentioned as OldTCP and OldUDP). Then I needed to split OldTCP and OldUDP layers from IP layer so that we can use our TCP and UDP classes. Here is the content of new TCP class’s guess_payload_class function;

def guess_payload_class(self, payload):
    for sig, cls in tcp_signatures:
        if re.match(sig, payload, re.IGNORECASE):
            return cls
# if signatures are empty or signatures matched
return OldTCP.guess_payload_class(self, payload)

where the tcp_signatures is a simple array of (signature, class) tuples;

tcp_signatures = [
    (r'^(EHLO|DATA|AUTH|MAIL|RCPT|QUIT).*', SMTPRequest),
    (r'^(GET|HEAD|POST|OPTIONS|PUT|DELETE|TRACE|CONNECT).*', HTTPRequest),
    (r'^(HTTP\/[0-9]).*', HTTPResponse),
    (r'^(230|331).*', FTPResponse)
        ]

In order to add a new TCP protocol write your new Packet class and add with it’s signature to tcp_signatures array, that’s all. If you want to check the code: https://github.com/honeynet/ovizart-ng/blob/gurcan-devel/tagger/protocol/__init__.py

Network Analyzer Project Updates – Week 4

This week I was not able to do a lot, because of my MSc. thesis defence seminar. Anyway, this week Mostly, I read and searched about the deep packet inspection and how we should do it. I found very nice papers about it and I will list them in wiki. They may provide us some ideas for further improvements.

Currently we have 2 choices about how we detect the protocol of streams in given pcap files, packet-based signatures and stream-based signatures. Both of them are basic regexp searches, they differs on what data they are applied.

Packet based signatures are easy to implement but also easy to bypass. Checking each packet payloads against the signatures works well against non-fragmented and (especially) unencrypted (protocol’s) packets. On the other hand we can search for stream-based signatures after reassembly of whole traffic, which gives us more information about the whole traffic communication, whether it is encrypted or not. This approach would be more accurate but would be slower as well.

For now we choose to implement packet-based signatures but later we will add stream-based signature and use on streams where packet-based signatures fails to detect application layer protocol.