Category Archives: Project 11 – Improving HPFeedsVisualization

Project11 – Improving HPFeedsVisualization-Week4/5

This two weeks , I completed a translator between HeliosJS and D3.JS.It connects them through callback function. HeliosJS will serve as the main tool for graph operations, on the other hand, D3JS will be Visualization as our main tool. I will put these two tools into Kibana in the future.

In the backend side,We decided to extract items from the thug events log and thug files log. Those items are as like follow:

thug.file logs

thug.event logs

We use a parser to extract those items, and then import them into Logstash. We will use these item to build the relationship between them. As like below graph



Project11 – Improving HPFeedsVisualization-Week 3

In week 3,We installed and learned Logstash,Elasticsearch and Kibana.
Logstash is a tool for managing our logs.We use it to receive logs from hpfeeds.
It extracted the objects from logs while stored those objects as well as indexing.
Those objects are such as time,hostname,url,and md5 value of malware in those logs.
Elasticsearch is flexible and powerful open source, distributed real-time
search and analytics engine for the cloud.In here,we used it to search those logs after logstash processed.
Finally,Kibana is a highly scalable interface for Logstash and ElasticSearch that allows our to efficiently search, graph, analyze and otherwise make sense of a mountain of logs.We have completed the installation and start running all above.
In the future we will base on Kibana interface to provide application and achieve our goals.
The first step will be a combination of Kibana,HeliosJS,and D3.JS.



Project11 – Improving HPFeedsVisualization-Week 2

In week 2 ,I discussed the goals in this project with Julia as follow:
First of all we will extract objects which from thug event logs or thug file logs as like url,hostname,and malware
to build the transmission graph.The relationships between malicious web pages and malware will be presented in the
The second,we will provide a function is to observe the evolution of transmission graph.
The third,Because we are interested in which node is the more activity,we should design an algorithm to score
activity in these nodes.
Finally,we could summarize the relationships to form several intentions,and should simplified the transmission graph
into the intention grpah.
On the other hand,We decided to change the system platform,because splunk has a lot of restrictions on development.
Initial system architecture is as follows:



Project11 – Improving HPFeedsVisualization-Week 1

Hi everyone,
I am Vincent ,and I am happy to start this project.
The progress of the first week as follow:

(1)Familiar with the system architecture and test environment

First,It took me some time to understand the system architecture and related technologies like hpfeeds ,splunk,and d3.js etc …

(2) Learning HeliosJS

In order to improve the readability of HpfeedsHoneyGraph , We believe that is necessary to simplify the HpfeedsHoneyGraph.So we should use some of graph traversal algorithms to detect objects which will be simplified.HeliosJS is an in-memory graph database for modern browsers, and it support efficient functions to help us to traverse the graph.GraphSON is a JSON-based format for individual graph elements and it was used in HeliosJS as well as D3.js. But there are difference default attribute name between HeliosJS and D3.js An example as follow:
heliosJS json attributes
vertices attributes:
1.”name” name of node
2. “_type” type attribute is vertex or edge
3.”_id” unique identifier
edges attributes:
1.”_id” unique identifier
2.”_outV” each edge has an outgoing tail vertex.
3.”_inV” each edge has an incoming head vertex.
4.”_type” type attribute is vertex or edge

d3 graph json format
vertices attributes:
1.”name” map heliosJS vertices attribute name
2.”id” map heliosJS vertices attribute _id
edges attributes:
1.”source” map heliosJS vertices attribute _outV
2.”target” map heliosJS vertices attribute _inV
3.”id” map heliosJS vertices attribute _id

Therefore, We will define the attributes for each nodes and links in the future.

Next Week:

(1) define the attributes for each nodes and links
(2) finish “events formalization” according to my proposal