Docker is Ultimately a Good Idea

So, over the past several months I have been using docker heavily. I started out with just plain vanilla Docker and then moved to primarily using Compose. This journey has been incredibly eye opening for me and I have come to the conclusion that Docker is ultimately a good thing. Here's my take:

The Good

Isolated environments without the cost. Absolutely beautiful. One of my greatest pains in building and maintaining applications is when you need to productionize and scale the app and you have to remember every package and config you needed installed. Sure there is Chef and Puppet which are great, but there is a certain discipline to the Dockerfile. I have taken it on myself to always build everything from the Dockerfile, and that discipline ensures you always have everything you need. For a guy who has 4-6 projects on his plate at a time, this is invaluable to me.

Rapid prototyping is hugely beneficial. I know the term 'ninja' is horribly overused today, but I really do feel like a ninja with Compose. The ability to spin up whole ecosystems in such a short amount of time has become very empowering for me. When looking for the right tool for the job, it really pays to be able to spin up different technologies quickly. I have gotten huge wins out of this in my current role. There is just nothing quite like seeing a tool in action before deciding on it, and Compose enables me to do that quickly.

The Bad

Increasing complexity. Docker ultimately adds a level of complexity in places. I always say that when you simplify one thing you are ultimately complicating another. This holds true with Docker. It is its own init system in a lot of ways, but then totally lacking in others. The logging situation can be difficult, networking can make your head hurt, and sometimes producing environments with minimalist kernels can have unexpected results. I have definitely put my head through a wall many times with Docker. It has great ambitions, but there are still gaps that need to be filled.

Microservices are not always good. With Docker inevitably comes the mantra of microservices. Personally, I really like the idea for a lot of stuff, but it is ultimately a dogma. The first application I built with Docker I adhered strictly to this doctrine. The service was really cool in the end and taught me a lot, but looking back I think it just took a lot more time than if I had allowed for more than one process per container. I think we will end up in a land somewhere in-between microservices and monoliths, I am heading that way now and its been good so far.

The Future

Fundamentally, I see Docker as solving the problem of process management that unix failed to. I think the energy just moved out of that world, probably because its not all that attractive from a top level. This left room for a new init on the block. With a rework this fundamental there has been a tremendous amount of effort poured back into this area. However, we are beginning to see a bit of a plateau in that today, and a lot of the cracks are beginning to get more attention. The Docker 'haters' are growing in numbers by the day, but I'm going to ice that fire a bit.

Docker is the future. I know people are probably tired of hearing that, but it's true. I imagine a reimagined hypervisor will return and microkernels will pick up steam, but at the heart of that movement is still the Docker brand. The are making smart acquisitions and seem well aware of the shortcomings of the current platform. I believe in the positives of the product, and see a lot of the negatives as solvable. So for now I'm sticking with the whale, full steam toward the containerization of everything.

Docker, Vagrant, Unix, Python, and Data Science

I haven't written in a while so I thought I would do a quick update post on what I've been working on.

Unix

I have dove deeper into Unix on a number of fronts. I have always been a Centos man, and recently discovered new ways of using Unix. First, I have learned Vim in recent months, and the experience has been very positive. I love being able to fly though my servers, editing code quickly and jumping to the next. I feel a lot of power using it, and I have officially given up on the IDE. I have built myself a nice vim setup and have it all linked up to Github so I can set it up quickly. Here's a nice screenshot

I have also just upped my terminal skills in general. I am really getting a hang of the whole Unix piping idea, and I am having a lot of fun with it. I have been writing some basic Python system level scripts for map/reduce. I am feeling that I may like Unix a lot more than the JVM. I feel free on Unix, and I am really beginning to understand how useful the limitless customization is.

Vagrant Up

As much as I was beginning to really like Unix, I was also running into a lot of issues. I needed different environments for different things, and it was a real pain to set up each time. A buddy of mine turned me on to Vagrant, and I quickly fell in love. Being able to quickly whip up VM's has been a godsend. I now have several boxes that are already set up with all the basics I need. I have functional environment in Centos6, Centos7, and Ubuntu14.04. Each one is set up with Vim and a custom terminal so I can quickly tell them apart. I had been running into the issue of working on the wrong server, because I had lost track. This allows me to easily separate them, they all have custom colors and log on messages. Here's an example:

This has spawned me to create a new repository on Github in which I am just making cloud machines. I call it Machimo, and the basic premise is that I break my development cycle into several machines that can be reused in different apps. So, I have a machine that just does API and exposing things to the web. A machine that just does Data Science with all the libraries preconfigured, and system optimized. A machine that just persists data to databases, and a machine that just runs the databases. Finally, A machine that just scrapes for data using proxies. I am really taking to this idea and I think it's a lot more manageable. I am frequently jumping between working on my own app and trying out things with friends, so it's nice to be able to reuse these things within a component framework. Here is the link to the github.

Docker

So, if you know Vagrant, you now at least know of Docker. It's containers are just VM's but lighter. I have been plain around with these as well and we have a bit of a love/hate relationship. On one hand they are wonderfully light, which makes them much easier to share than the Vagrant package route. The ability to pull then down in production is really nice, and being able to just replicate instances so quickly makes my imagination spin. However, there are a couple drawbacks. First, I'm not sure that it's ready yet. I do know some people using it in production now, and they have expressed some concerns to me. Also, it seems that the documentation isn't really up to date. I had a little trouble getting it going, the official docs left quite a bit to be desired, and it seems they have changed their libraries substantially over versions so much of what you find on Stack Exchange isn't really valid.

I am excited about this technology and think it has a lot of potential. However, I am at a place right now where I'm not wanting to wrestle with low level stuff. I just got Vagrant the way I like it and it works well enough for me. I am more focused on Data Science right now and just don't have as big a need for it. I can see myself using it in the future, but I think I'll wait a year or so before jumping back in.

Python! Data Science!

After spending a couple years learning the basics of development I was hungry for something more. I began to understand how things interacted and became proficient in building applications. However, I came to realize that the questions in my head required I give deeper meaning to the data that I was passing around. I wanted to model things in multiple dimensions. I wanted to quantify the movement of ideas and interactions in cultures. So, I began on my path towards data science.

There are a couple languages you can go with here. I know Scala already and it is a decent choice for big data, but Python and R stick out the most. R seems to be more focused on pure analytics and mathematics, whereas Python is general purpose. It has state of the art math and science libraries, but can also be used in almost all other use cases. So, because I am interested in building intelligent applications, Python seemed to be the right choice for me, and I have been very happy with it so far.

What have I seen in Python? Amazing libraries, I mean really amazing, and they all seem to have excellent documentation. Cool features like list comprehensions. While I like Scala's syntax a little more than Python, there are some really neat things in Python like these comprehensions:

Also, IPython is nothing short of amazing. I have fallen in love with this idea. I can prototype so faster than ever before, and I feel like I can reach an understanding of topics and processes quicker by using it. It looks I am going to be sticking with Python for awhile, it seems like the perfect language for this part of my life.

In the data science direction, I have taken some probability and statistics classes recently. Also, I worked through the Data Science in Python O'Reilly book. And now I have been working through the book and online class Coding the Matrix, which teaches linear algebra in Python. I have really been enjoying it. I know I'm heading the right direction, because it just seems like fun doing it.

Till next time

Getting Started with Titan Graph, Gremlin-Scala, and Play!

Hello Graph

So, I've been trying out different graph databases for my next project. I really like the way that they organize things, it seems to match my thinking quite a bit. I wanted to stay open source so I skipped neo4j, due to its licensing. I first tried OrientDB, which was nice, but then I came across Titan, a distributed graph db. Titan is built on top of your choice of HDFS, you can use BerkelyDB, Cassandra, or HBase. Berkeley is more for small-medium graphs, but it performs much faster because it is local. Whereas, Cassandra and Hbase can be distributed across a cluster and hold massive graphs. For my use case I went with Berkeley, because it's fast and I don't think I'll approach its ceiling with my next app.

Hacking Niagara AX with Scala!

Integration can be a pain

 If you are in the automation world, you know how frustrating it can be to integrate to old systems. Last week I was given the project of cutting over an ALX system in a 10 story building. After exposing all of the points on the ALX system by hand, I had enough. I had recently discovered that all of Niagara's folders could be viewed as XML. This is super handy for making easy transformations like find "x" and replace with "y". Any basic text editor has this capability. However, I needed more power. In order to automate my task I needed complex transformations. The kind you need to write an algorithm for.