Live Bubble Script
with GSoC coming to an end for this year, let’s have a look at what we have accomplished over the last three months:
- Integration into mitmproxy
- User Interface redesigned from the ground up.
- Completely overhauled traffic table, supporting up to a million (lazy-loading) rows.
- New Filtering Syntax for searching flows (inherited from mitmproxy).
- Improved Report Editor
- DNSChef Integration
- Live Bubble Report Script
- Huge internal refactorings: HoneyProxys inner architecture is prepared for the future!
- Support for multiple scripts and script arguments in mitmproxy
- Standalone Windows Executable.
To sum it up, I’m really happy with the progress we made. Lots of exciting things made it into mitmproxy.
New Interface with searchbar and filtering syntax
So, when can we expect the next release? Long story short, there are still a few rough edges that will be ironed out as soon as @cortesi has completed the sqlite integration. He has been pretty busy over the last few months and we couldn’t complete this within the GSoC timeline, but I’m commited to continue my work on
HoneyProxy mitmproxy as I did last year.
Showing a DNS request with DNSChef
Overall, this was an exciting summer again and I’d like to thank everyone who helped over the course. There are four guys who deserve special attribution:
Aldo, thank you for patiently answering all my mails and the good discussions.
It’s a pleasure to work with you. :-)
Guillaume and Sebastien, thanks for your great mentoring.
David, thank you for organizing this HoneyNet GSoC again – you’re doing an invaluable job here.
Let’s keep on coding!
PS: You like living on the bleeding-edge? Check out our dev snapshots and the GitHub repo!
It has been pretty quiet on HoneyProxy lately, but I’m happy to say that we’re right on track for the GSoC schedule. I polished the DNSChef integration since the last blog post, employing the new scripting interface I developed earlier. You can now use DNSChef (or any other mitmproxy script) with its own custom parameters from the command line. For example, you can configure the DNS port using DNSChefs
honeyproxy.py -s "dnschef.py -p 5353"
While this is just a minor addition, it’s the most visible one of the work of the last few weeks at the moment. Most resources have been spent on the mitmproxy integration, filling issues, moving code between repos, fighting git submodules etc. I’m optimistic that we can finish this within the next two to three weeks. In contrast to the completed fraction, there will be some noteworthy architectural changes over the next weeks, so expect more blog posts about this soon!
Here we go for status report #4 :). As already predicted last week, my exams eat my GSoC time and development gets slower over the next few weeks. Nevertheless, we made some good progress this week I’d like to outline briefly.
First of all, we have an experimental DNSChef integration! While there are still some minor issues left, it already benefits from improvements we made earlier this GSoC, namely support for script arguments. For the next 1-2 weeks, I’m going to fix the remaining issues and take care of a detailed documentation.
Secondly, Aldo and I discussed that we’re tackling the mitmproxy integration next (before migrating to SQLite), so I’m happy to share that the HoneyProxy codebase will be part of mitmproxy sooner than expected.
Time flies… here comes the status report for the third week of GSoC!
This week I completed two distinct building blocks – a new search bar and live updates for the bubble reporting script.
The searchbar replaces the old search functionality in the right sidebar. It frees a lot of space and provides a superior usability compared to the old implementation. It consists of a filter input (on the left) and up to 20 highlight inputs. While the filter removes all rows from the view that don’t match, the highlighters color the matching rows and preserve the others. In the screenshot you see the orange highlighter in the middle, with the next highlighter indicated on the right – new highlighter inputs get added dynamically as needed.
The added live functionality for the bubble script looks really fancy. Some Honeynetlers may know the general concept from Dubai already, but let’s give you a quick recap: Each captured host gets displayed as a node, with communication between host A and B displayed as a link (e.g. in the example you see that the blue node communicates with six isolated nodes that don’t talk to each other [at least not in our observations]). Hovering over the nodes reveals their hostname, yellow blinking edeges indicate a live request. The request frequency is indicated by the link color (grey to red), the last activity time from a host by its node opacity. The node size is determined by the amount of data that has been sent. Taking a look at the example again, you might assume that the blue node is a browser sending HTTP requests without content (let’s say GET) and receiving larger response contents in return. Later in the animation, its size increases as it does a large POST request to the dark green node.
For the next week, our goal is to add DNSChef support as a mitmproxy module. The following two weeks will be less productive due to my exams, but we’ll catch up after that as we did last year.
The second week of GSoC is over and I’m happy to say that it has been productive as well. As Aldo is currently busy with his work, I spent this week directly working on HoneyProxy, designing and implementing the client-side Traffic Store for flows. As a quick recap, HoneyProxy currently stores all captured flows in memory. While this makes processing easier, we cannot analyze large datasets (the current cap is at approx. around 10.000 flows). One of our goals this summer is to tear down this barrier by employing a lazy-loading dgrid on the client side and providing querying capability on the server side.
The new flow store is conceptually based on dojos Real Time Stores with custom observation logic. It turns out that observing datasets where (a) the order of rows can change, (b) new rows are added dynamically and (c) all querying logic is bound to the server is a non-trivial task. For example, while we can observe that new flows are captured, we still don’t know whether they match the current client-side filter and should be added to the view. Also, we can’t be sure that they are added at the end of the list (e.g. sorted by response size) and need to adjust the position of other flows as well. Our data store accounts for all these difficulties by requerying small fractions of the dataset from the server and comparing them to cached results.
Unfortunately, a correct Store implementation combined with our extreme use-case exposes some nasty bugs in dgrid. I spent the rest of the week working on these issues and tackled all of them. At the end, I can confidently say that we reached our goals for this week and added a powerful Store implementation for HoneyProxy, allowing us to process large data-sets on the client-side.
I am Max and I am continuing the development of HoneyProxy, my GSoC project I started last year. I hope everyone is going to have as much fun as I had last year.
The first week of GSoC is over and we already have some great news to share: I am super happy to announce publicly that HoneyProxy will be integrated into mitmproxy this summer. In fact, I am joining Aldo Cortesis mitmproxy project as the first external core developer, bringing the existing HoneyProxy codebase with me. Working with Aldo has been a really pleasant experience over the last year and I am confident that we’ll see some nice results of our collaboration at the end of the summer. Merging two projects always requires much effort, so both the Community Bonding period and the first week have been full of coordination and organisational tasks. In spirit of GSoCs openness, you can see a brief overview of all the things that needed consideration here. Nevertheless, a week without code is a bad week, so we already have the first code results as well:
- mitmproxy: add support for multiple scripts and script arguments (2b4af8d)
- mitmproxy: improve windows compatibility (ca9740df20, ba47690a03, …).
- mitmproxy: fix a really nasty (and difficult to find) bug in netlibs tcp handling (68e2e78)
- mitmproxy: other minor bug fixes
- HoneyProxy: New Traffic Table based on dgrid (resizable columns, selectable columns, virtual paging, …) (branch:dojo_store)
- HoneyProxy: Bootstrapped Interface
- HoneyProxy: Huge internal refactorings
- HoneyProxy: Add Template Widget does fancy data-binding
Note: Some code (especially on HoneyProxy) has already been written during the community bonding period and is thereby no official part of GSoC. I figured out it’s worthwhile to include it here nevertheless.