This two weeks , I completed a translator between HeliosJS and D3.JS.It connects them through callback function. HeliosJS will serve as the main tool for graph operations, on the other hand, D3JS will be Visualization as our main tool. I will put these two tools into Kibana in the future.
In the backend side,We decided to extract items from the thug events log and thug files log. Those items are as like follow:
We use a parser to extract those items, and then import them into Logstash. We will use these item to build the relationship between them. As like below graph
In week 3,We installed and learned Logstash,Elasticsearch and Kibana.
Logstash is a tool for managing our logs.We use it to receive logs from hpfeeds.
It extracted the objects from logs while stored those objects as well as indexing.
Those objects are such as time,hostname,url,and md5 value of malware in those logs.
Elasticsearch is flexible and powerful open source, distributed real-time
search and analytics engine for the cloud.In here,we used it to search those logs after logstash processed.
Finally,Kibana is a highly scalable interface for Logstash and ElasticSearch that allows our to efficiently search, graph, analyze and otherwise make sense of a mountain of logs.We have completed the installation and start running all above.
In the future we will base on Kibana interface to provide application and achieve our goals.
The first step will be a combination of Kibana,HeliosJS,and D3.JS.
In week 2 ,I discussed the goals in this project with Julia as follow:
First of all we will extract objects which from thug event logs or thug file logs as like url,hostname,and malware
to build the transmission graph.The relationships between malicious web pages and malware will be presented in the
The second,we will provide a function is to observe the evolution of transmission graph.
The third,Because we are interested in which node is the more activity,we should design an algorithm to score
activity in these nodes.
Finally,we could summarize the relationships to form several intentions,and should simplified the transmission graph
into the intention grpah.
On the other hand,We decided to change the system platform,because splunk has a lot of restrictions on development.
Initial system architecture is as follows:
I am Vincent ,and I am happy to start this project.
The progress of the first week as follow:
(1)Familiar with the system architecture and test environment
First,It took me some time to understand the system architecture and related technologies like hpfeeds ,splunk,and d3.js etc …
(2) Learning HeliosJS
In order to improve the readability of HpfeedsHoneyGraph , We believe that is necessary to simplify the HpfeedsHoneyGraph.So we should use some of graph traversal algorithms to detect objects which will be simplified.HeliosJS is an in-memory graph database for modern browsers, and it support efficient functions to help us to traverse the graph.GraphSON is a JSON-based format for individual graph elements and it was used in HeliosJS as well as D3.js. But there are difference default attribute name between HeliosJS and D3.js An example as follow:
heliosJS json attributes
1.”name” name of node
2. “_type” type attribute is vertex or edge
3.”_id” unique identifier
1.”_id” unique identifier
2.”_outV” each edge has an outgoing tail vertex.
3.”_inV” each edge has an incoming head vertex.
4.”_type” type attribute is vertex or edge
d3 graph json format
1.”name” map heliosJS vertices attribute name
2.”id” map heliosJS vertices attribute _id
1.”source” map heliosJS vertices attribute _outV
2.”target” map heliosJS vertices attribute _inV
3.”id” map heliosJS vertices attribute _id
Therefore, We will define the attributes for each nodes and links in the future.
(1) define the attributes for each nodes and links
(2) finish “events formalization” according to my proposal