Primary databases.. Why not use Redis?

* (A link to the code is available at the bottom)

While I was beginning to develop my latest personal project I was thinking about which database to use. Two years or so ago, I would have just defaulted to MySQL. It was what I was familiar with, and I’d never really had an issue with it. But then, the inevitable happened. I built apps/websites that actually had decent traffic, or ran on servers with extremely limited resources. And go figure, MySQL was a nightmare. So for this latest project, that left me with the question: Which database should I use?

Now in real life, in my opinion, the answer is “it doesn’t matter.” By this I mean, show me a company that failed solely because of the tech stack they chose. You might be able to find one or two tiny ones but in reality, it just doesn’t happen. However, as an engineer you get to deal with the concept of “Technical Debt”. Which means that you should plan to the best of your abilities, to avoid having to fight it or fix it later on down the road. But that still left me with A LOT of choices. MySQL, Postgres, MongoDB, Riak, …. and Redis?

My requirements were pretty basic. I needed the fastest possible write speeds, with reasonable select speeds. The data itself is incredibly basic. There are two types and they’re associatively related to each other. So I got to thinking about it, and I decided to create a Redis model that implemented a basic Collection design pattern. All of this works pretty well, but there were a couple challenges that I had to face.

Really there were two primary challenges. Those being: Searching (you can only search keys in Redis) and auto-incrementation (Redis has no support for this). So tackling the easiest, (and necessary) one first, auto-incrementation. This one was simple, it takes the lowercase pluralized version of the model name, and created a key with a basic integer value. The integer is increased every time a resource is created, and it’s never decreased. Making it so you could GET resource_name and use that +1 for your new resources ID – which happens automatically. The second challenge was a little more complicated. How do you search for a specific resource, if you can only search by keys? For a User model, you need at the very least to be able to search by username and password.

To accomplish this task, I decided to have a key, with an underscore delimited list of the searchable indexes. That allowed me to use KEYS with the given columns, using some kind of format like user:username_hashed-password which would return the ID of the resource. Or, user:*@gmail.com_* which would return a list of IDs. This ended up working really well. Until I realized the user could literally enter admin_username, * and it would match, since it’s all the same to Redis. Which created the need for two different methods: searchBy and searchByWildcard. The underlying code is the same, since searchBy passes the request along to searchByWildcard AFTER it strips out all uses of the character *. This is akin to using mysql_real_escape_string() in pre-PDO PHP days. Which solved that problem.

In conclusion, the Model works really well. And I can definitely see use-cases for it. However, my use-case is not one of them. I found that once there were tens of thousands of rows, iterating over the keys got unacceptably slow when performing a login since I had other searchable indexes besides just username and password. But you could take the same idea, and optimize it a lot by implementing specific searchable pairs, and then for basic datasets virtually remove the need for ever iterating over keys. So I built it, and it’s pretty cool, but I ultimately ended up switching to PostgreSQL and continuing to think about my next project which could very well incorporate my new RedisModel data accessor.

Link to the code & the Model if you want to implement it into your project:

Ideas? I have dozens of those…

I come up with new ideas for services, apps or businesses all the time, almost every day. Now, obviously I can’t do them all. There just isn’t that many hours in a day. So how do I decide which ones to pursue? I guess there isn’t necessarily any rhyme or reason to it, at least, not a lot. Mostly it just has to do with how clearly I can see the entire roadmap of the idea. For example, sometimes I think of an idea, which sounds great, but after I think about it a little more I realize there isn’t a very good way to execute it, given my current situation.

For example: I have a game that I’ve wanted to ‘play’ for years. The play part is key here, and we’ll get back to that in a minute. The idea behind the game is that the better you get in real-life, the better you get in the game. So if you get stronger in real-life, you get stronger in the game. Same goes with the rest of the skills or levels in the game (as far as they make sense, obviously I don’t envision people actually mining, or forging things out of metal). Now, a game like this doesn’t exist. Since I want to play it, I can guarantee others would want to play it too. Which is the key to a good idea, if it’s an issue or a want in your life, there is a good chance that it’s an issue or problem in someone else’s too.

Now, occasionally I pursue various ideas for this game or maybe a more toned down version. Or even sometimes I get inspired about a story-line that could fit well with the game. When I get inspired, I don’t argue with myself, I just work on it. I have dozens of files, and multiple different version, languages & concepts all relating to ‘The Game’. I’ve given it different names at different times, and it’s had a few different premises, but the core dream has never changed. But why don’t I just pursue it for 6 months and get a beta version working? Simple, I’m not in a position currently to do that.

What I mean is, there are other things more important to me. There are other things I want to build. There are also skills required to make the game that A) I don’t have, B) I don’t want. In order to truly accomplish what I want, it would have to be made using some kind of Virtual Reality system. I’m not a graphics designer or graphics programmer. I can do it, but it would take me years to get to the point of being able to design & implement a game in a VR setting. But if I could hire a team of developers & artists to help me accomplish my dream, I would. That is where planning out ideas comes in.

Since I can’t hire a team of people right now, I’ll work on ideas that I can actually get to ‘shippable’ by myself, or with 1 other person. Though this sounds easy, I assure it is not. It still requires hours of work, and thought and methodical planning. Plus, outside of just creating it, you need a marketing plan. Otherwise you just spent a ton of time, to not make any money. There are 2 trains of thought that I have come across as far as ‘when to market’ goes.

One train of thought, is if the product isn’t perfect, marketing it and having people finding bugs/errors/flaws with it will kill your company. The second one is, push fast, fix faster. The idea behind the second train of thought is (especially in terms of a web service) nothing is permanent. You can fix the issues faster, if you know what they are. If no-one is using features D & E of your service, why fix the bugs in them? If people are using feature B in a way you didn’t originally intend, why spend a few dozen hours making it perfect, just to make a 90 degree turn to meet the actual needs of the end-user?

I can’t necessarily say which one is better. Both have been done in successful products. What I can tell you, is having a marketing PLAN is never a bad thing. You don’t need a product to have a plan. You only need an idea to have a plan. If you can’t see the entire road for taking your idea from a thought, to a profitable business, then the likelihood of it ever getting done is almost non-existent. You can think about a plan the entire time your designing, creating & building.

So in conclusion, how do I decide which ideas to pursue? The ones where I can clearly see how it will become a profitable business. How I can build it, design it and market it. If I can’t see all of those things, usually it doesn’t even stay in my head that long, because it usually means there is a core disconnect between the problem-scope that the idea solves, and the solution I thought of.