forensicBlend: Designing a scalable community plugin API
I decided to start writing this series to document my work on forensicBlend, a project I previewed on Twitter yesterday that takes device logs and translates them into a modern report format that can be searched, filtered, and exported. One of my fundamental design goals is to provide a high level of extensibility and allow community developers to contribute. That is, I want for people who know how to script to be able to contribute their own custom logic and see it work in my apps (and ultimately, result in a better timeline tool). This is something Eric Zimmerman touched on in our interview last week.
I scaffolded some of the UX (above) based roughly on how I want this to work. Essentially- list the currently installed plugins, prompt for updates where available, and provide a way to browse the online repository to download additional plugins.
Side note: I’ve had some questions regarding what UI framework I am using. The above is a WPF app which uses the excellent MahApps.Metro and Material Design in XAML libraries. These are free offerings that you can use in your own WPF project to elevate your UI to the next level.
There are really two areas of work here to think about: the plugin API itself (what to do with the packages once they are installed), and package hosting/redistribution. Here are a few overall design considerations and requirements I came up with:
- As a lone developer, time and cost savings are a priority. If there are any wheels that have already been invented, don’t invent new ones unless the need be great.
- It’s 2019 and it’s therefore important we take time to consider things like security. Packages containing plugins may have DLLs (more on this later) with code that will ultimately be executed by our app. This could (and should) be considered a potential attack surface. We can mitigate this with some of the following:
- Packages should be signed and verified at every step of the way.
- Community created plugins and updates will undergo a thorough, manual code review and testing before they are posted to the package library (think Apple’s App Store).
- Plugin packages should have versioning capabilities and upgrading to the latest version should be as seamless as possible, being cognizant that not all users will have internet access.
- Community created plugins will be managed centrally and approved prior to being posted to the online package library (think Apple App Store).
Hosting and Package Distribution
I knew going into a project like this that I wanted to use Amazon AWS. The cost effectiveness, sheer scalability, and all-around cool factor of using AWS made this an easy design decision. Off the top of my head, I expect to be using the following AWS components:
- API Gateway
- Create and administer web endpoints for the app.
- Certificate Manager
- Free SSL certificate!
- Content Delivery Network (CDN) to ensure low latency, high speed access to data from anywhere in the world
- Complete User Account Management and Authentication
- EC2 (Elastic Compute Cloud)
- Host microinstance of some sort of RDBMS. Or perhaps we will try out a NoSQL solution DynamoDB for science / learning — and because DynamoDB has a permanent free tier option.
- Elastic Load Balancing
- Distribute incoming application traffic across multiple targets across several Availability zones.
- Provide the business logic for serving our REST API to answer questions like “What are all the plugins currently available and what is the latest version?”
- Provide the business logic for facilitating and monitoring package downloads (users like to see download counts), potentially provide a ‘thumbs up / thumbs down’ interaction or possibly even comment
- Route 53
- DNS Registration
- Secure, encrypted, redundant hosting of the compiled packages themselves
- Web front-end for users who choose to browse it this way.
Let’s keep in mind I’ve touched on less than 10 of the things AWS can do for you, whereas the actual list of things it can do is much, much longer. For most of us, the usage involved will be in or around the Free Tier, so basically what I’m saying is you can get all of the things above for NO COST. If you are reading this and going, “Why am I still renting web space like I did in 2005?” this is an excellent question. You may wanna migrate! There is also an irreplaceable feeling you get when you realize you are using the same exact same serverless environment as some of the largest of corporate juggernauts out there.
Earlier I mentioned leveraging as many existing technologies as possible. Most every .NET programmer out there is familiar with the idea of NuGet. From NuGet themselves:
NuGet is the package manager for .NET. The NuGet client tools provide the ability to produce and consume packages. The NuGet Gallery is the central package repository used by all package authors and consumers.
So if I’m writing an app, and I want to bring in code from a library to perform a specific function, I can open the NuGet Package Manager in my development environment (Visual Studio shown below) and perform a search for the function I need. Then it’s one click to install and be off.
It goes even further by providing strong versioning, licensing, dependency tracking, and more. Behind the scenes, NuGet uses .NUPKG files which are containers that bake alot of this functionality in for us, and provide desirable things like package signing.
Since NuGet already does everything we need and then some, for free, why would we design our own solution from scratch? Seeing the recurring theme here?
So we know we’re going to use NuGet as a package management solution, and we know we’re going to use AWS for community hosting and package distribution, but what about the actual code to load said plugins? This could be the most entertaining part, but also the most time intensive. For the purposes of development, I will need to look at how to extract and use content from a NuGet package at runtime.
I’m going to have to weigh the advantages of dynamic code compilation (source code compilation at runtime) vs. distributing pre-compiled binaries (DLLs) and simply loading them.
Stay tuned for the next article in the series where we will get coding!