Primary databases.. Why not use Redis?

* (A link to the code is available at the bottom)

While I was beginning to develop my latest personal project I was thinking about which database to use. Two years or so ago, I would have just defaulted to MySQL. It was what I was familiar with, and I’d never really had an issue with it. But then, the inevitable happened. I built apps/websites that actually had decent traffic, or ran on servers with extremely limited resources. And go figure, MySQL was a nightmare. So for this latest project, that left me with the question: Which database should I use?

Now in real life, in my opinion, the answer is “it doesn’t matter.” By this I mean, show me a company that failed solely because of the tech stack they chose. You might be able to find one or two tiny ones but in reality, it just doesn’t happen. However, as an engineer you get to deal with the concept of “Technical Debt”. Which means that you should plan to the best of your abilities, to avoid having to fight it or fix it later on down the road. But that still left me with A LOT of choices. MySQL, Postgres, MongoDB, Riak, …. and Redis?

My requirements were pretty basic. I needed the fastest possible write speeds, with reasonable select speeds. The data itself is incredibly basic. There are two types and they’re associatively related to each other. So I got to thinking about it, and I decided to create a Redis model that implemented a basic Collection design pattern. All of this works pretty well, but there were a couple challenges that I had to face.

Really there were two primary challenges. Those being: Searching (you can only search keys in Redis) and auto-incrementation (Redis has no support for this). So tackling the easiest, (and necessary) one first, auto-incrementation. This one was simple, it takes the lowercase pluralized version of the model name, and created a key with a basic integer value. The integer is increased every time a resource is created, and it’s never decreased. Making it so you could GET resource_name and use that +1 for your new resources ID – which happens automatically. The second challenge was a little more complicated. How do you search for a specific resource, if you can only search by keys? For a User model, you need at the very least to be able to search by username and password.

To accomplish this task, I decided to have a key, with an underscore delimited list of the searchable indexes. That allowed me to use KEYS with the given columns, using some kind of format like user:username_hashed-password which would return the ID of the resource. Or, user:*@gmail.com_* which would return a list of IDs. This ended up working really well. Until I realized the user could literally enter admin_username, * and it would match, since it’s all the same to Redis. Which created the need for two different methods: searchBy and searchByWildcard. The underlying code is the same, since searchBy passes the request along to searchByWildcard AFTER it strips out all uses of the character *. This is akin to using mysql_real_escape_string() in pre-PDO PHP days. Which solved that problem.

In conclusion, the Model works really well. And I can definitely see use-cases for it. However, my use-case is not one of them. I found that once there were tens of thousands of rows, iterating over the keys got unacceptably slow when performing a login since I had other searchable indexes besides just username and password. But you could take the same idea, and optimize it a lot by implementing specific searchable pairs, and then for basic datasets virtually remove the need for ever iterating over keys. So I built it, and it’s pretty cool, but I ultimately ended up switching to PostgreSQL and continuing to think about my next project which could very well incorporate my new RedisModel data accessor.

Link to the code & the Model if you want to implement it into your project: https://github.com/Ryuske/RedisModel

How I Go From Idea to Product

How to take a new idea for a feature, website or app and turn it into an actual product? Well, for me, it’s a 5 step process. The more time I spend on the first two, the easier the rest is. None of it is complicated, but that doesn’t mean it’s always easy. The five steps are as follows:

  1. Clearly Outline the Goals
  2. Create a Clear Flow of How It Will Be Used
  3. Design the Interface
  4. Build the Implementation
  5. Make It Live (Market It or Push It)

Now these steps apply at all levels of the idea stage. You can use these 5 steps when building a brand new service from scratch, or adding a feature to your existing product. It can also be used to evaluate existing features or flows that you don’t feel are getting the conversions they should be.

For the rest of this article, we’re going to look at actually doing this for a feature. Let’s say, you and your team have decided that you need to build a voting system for the posts on your site. You could just wing it, fly by the seat of your pants and build something. And it could work, it could be fine. Or, you could follow these steps, and KNOW that it will work and it is the best implementation possible with the given information. That doesn’t mean it will be perfect, or that you’ll never have to change it. But those changes are hopefully driven by new data, not ‘just because’.

The first step, Clearly Outline the Goals. Sounds easy enough, but honestly, this is the hardest step of them all. What are the goals of a voting system then? Well, primarily I’d say so that your users can find the best posts written. So they can quickly see which post on a given topic has the most ‘bang for the buck’. Oh, this one only has 2 votes and it was published a year ago? It probably isn’t very good. But THIS one has 150 votes and was published 24 hours ago? I want to read that. Now that might seem like kind of a long goal, since we’re trained to want one-liners or catchy titles to pitch at business meetings – which is fine. But that doesn’t help you build the best implementation possible. Be verbose, use as many words as it takes to make it clear. If you give it to your 10 year daughter, does she understand what the point is behind the feature? If so, you’re probably good. If not, you should probably figure out what the problem is.

The next step, Creating a Flow for the feature. The amount of effort this takes largely depends on the scale of the idea. If you’re going through these steps for something that doesn’t exist, this will probably take a significant amount of time. Don’t half ass it. Draw it on paper, create flow charts, create a Power Point. Do all of it. Make it so that if you quit the day after finishing the first two steps, the next guy could pick up right where you left off, without having to call you. What does the flow look like? Well, there are two. The first one is as a voter, the second is as a reader. As a voter, I want to read the article, and then vote. So putting the vote buttons at the top of the article is probably not a very good idea. As a reader, I want the votes BEFORE I read the article. So don’t make the user click the article to see how many votes it has. Be forth-coming with information, you’re not selling something. Don’t bait & hook.

Now steps three and four can switch places depending on what you’re doing. If you’re not sure how to build it, and have no idea technically what it is going to take, start on the back-end, start on the implementation. As soon as you have a clear idea though, go back and Design It. Designing it and creating the flow can often times go hand in hand. Sometimes naturally a design will start to come in just based on the flow. But both steps are important to keep separate.

Designing the Interface, it’s really important to get creative here. I highly recommend starting on actual paper. Cut it all up, put it on a board, use strings to connect the dots. Do whatever you have to, but have fun & be creative. Once you finish that, go into your favorite image editor and design it all out. Now you can build a mock-up of it in HTML and make sure that it all translates well. The reason you start on a piece of paper is because it doesn’t take any effort to toss the paper to the side & start over. If you take the time to try and make everything on the computer, you tend to get attached to the first draft because it took effort to create, you became emotionally attached to the design. That is a good way to shoot yourself in the foot. So for our votes feature, we need a design for our thumbnail/excerpt that readers will see. Then we need to design the actual vote up (or down) buttons that are at the bottom of the post.

Building the Implementation by now you should have a clear idea of the flow, and exactly what it will look like. That should make building the implementation a snap. If the previous steps were done well, and as a collaborative effort within the team, that should drastically reduce the amount of time spent asking & answering questions. Having someone from each department involved (design, development, management, etc) as soon as possible in all of the steps will also help. It is frustrating for everyone involved when a design goes to development, just to have development say that the design isn’t feasible/possible. Eliminate this by making sure everyone is collaborating, and communicating their thoughts effectively.

The last step, Make It Live. This step depends again on what you’re doing with this exercise. If you’re adding a feature like our votes, then make it live and use funnel conversions to prove your feature works. Don’t just assume it works, do your best to prove changes. This is also a good mindset to get into when making changes. Prove there is a problem, and WHAT and WHERE the problems are. Don’t assume anything. If you’re using these steps for creating a whole new service, then this step is also marketing. Market the crap out of it. Be proud, tell the world about what you and your team just built. If you’re excited about it, then your end-users will be too.

Those are the 5 steps I typically take when planning & building things. It’s a lot to go over in a single blog post. I will probably do focused blogs on each of the steps at some point, walking through a real-life situation complete with pictures & code. But I will have to plan that out some more, and I have other topics I want to write about before then. So stayed tuned, lots of interesting things coming up!

 

Ideas? I have dozens of those…

I come up with new ideas for services, apps or businesses all the time, almost every day. Now, obviously I can’t do them all. There just isn’t that many hours in a day. So how do I decide which ones to pursue? I guess there isn’t necessarily any rhyme or reason to it, at least, not a lot. Mostly it just has to do with how clearly I can see the entire roadmap of the idea. For example, sometimes I think of an idea, which sounds great, but after I think about it a little more I realize there isn’t a very good way to execute it, given my current situation.

For example: I have a game that I’ve wanted to ‘play’ for years. The play part is key here, and we’ll get back to that in a minute. The idea behind the game is that the better you get in real-life, the better you get in the game. So if you get stronger in real-life, you get stronger in the game. Same goes with the rest of the skills or levels in the game (as far as they make sense, obviously I don’t envision people actually mining, or forging things out of metal). Now, a game like this doesn’t exist. Since I want to play it, I can guarantee others would want to play it too. Which is the key to a good idea, if it’s an issue or a want in your life, there is a good chance that it’s an issue or problem in someone else’s too.

Now, occasionally I pursue various ideas for this game or maybe a more toned down version. Or even sometimes I get inspired about a story-line that could fit well with the game. When I get inspired, I don’t argue with myself, I just work on it. I have dozens of files, and multiple different version, languages & concepts all relating to ‘The Game’. I’ve given it different names at different times, and it’s had a few different premises, but the core dream has never changed. But why don’t I just pursue it for 6 months and get a beta version working? Simple, I’m not in a position currently to do that.

What I mean is, there are other things more important to me. There are other things I want to build. There are also skills required to make the game that A) I don’t have, B) I don’t want. In order to truly accomplish what I want, it would have to be made using some kind of Virtual Reality system. I’m not a graphics designer or graphics programmer. I can do it, but it would take me years to get to the point of being able to design & implement a game in a VR setting. But if I could hire a team of developers & artists to help me accomplish my dream, I would. That is where planning out ideas comes in.

Since I can’t hire a team of people right now, I’ll work on ideas that I can actually get to ‘shippable’ by myself, or with 1 other person. Though this sounds easy, I assure it is not. It still requires hours of work, and thought and methodical planning. Plus, outside of just creating it, you need a marketing plan. Otherwise you just spent a ton of time, to not make any money. There are 2 trains of thought that I have come across as far as ‘when to market’ goes.

One train of thought, is if the product isn’t perfect, marketing it and having people finding bugs/errors/flaws with it will kill your company. The second one is, push fast, fix faster. The idea behind the second train of thought is (especially in terms of a web service) nothing is permanent. You can fix the issues faster, if you know what they are. If no-one is using features D & E of your service, why fix the bugs in them? If people are using feature B in a way you didn’t originally intend, why spend a few dozen hours making it perfect, just to make a 90 degree turn to meet the actual needs of the end-user?

I can’t necessarily say which one is better. Both have been done in successful products. What I can tell you, is having a marketing PLAN is never a bad thing. You don’t need a product to have a plan. You only need an idea to have a plan. If you can’t see the entire road for taking your idea from a thought, to a profitable business, then the likelihood of it ever getting done is almost non-existent. You can think about a plan the entire time your designing, creating & building.

So in conclusion, how do I decide which ideas to pursue? The ones where I can clearly see how it will become a profitable business. How I can build it, design it and market it. If I can’t see all of those things, usually it doesn’t even stay in my head that long, because it usually means there is a core disconnect between the problem-scope that the idea solves, and the solution I thought of.

SEO… Yourself?

I was reading one of Cap Watkins’ old blog posts (which you should definitely check out, along with his other writings), The end of self-curation the other day, and it really got me thinking.

I agree that ultimately the idea that this generations parents are probably not going to be keeping a box of pictures in the attic. And really, I think this blog post probably pertains more to people professionally than personally. But you never know.

So what then? In a world where your whole life and story are on social media, how do you make sure that people are finding out the parts of you that you really want them to? I remember back when I got my first “official” job as a developer, which was as an intern at DealerTrend, I was talking with one of my co-workers and the topic of “SEO’ing yourself” came up. Wait a minute. What the heck does that even mean? I don’t want to be found, why would I do that?

Look, it doesn’t matter if you want to be found, or not. If you’re on social media, the world can see you. Your family, your employer, and frankly your stalker. You may have all those fancy privacy settings turned on, but it doesn’t matter. You’re still out there, in some dark corner of the Internet, there is a section dedicated to you. Which means, you might as well embrace it, and SEO your online presence. Which, really means, make the things you want people to know easier to find, so they stop looking for the things you don’t want them to. But… How do I actually do that?

Well, lot’s of ways. One way, especially professionally, is to create your own website (or blog). And given how easy that is now-a-days, there isn’t really any excuse NOT to. You can also have social media accounts that you update regularly that are public, and have a 2nd one that just your friends on are. If someone finds your public one, they’re going to see what you want, and in all likelihood, stop looking for anything more. Even on a personal level, why not have one that is dedicated to just things important to you? That way when your kids are wondering who you were ‘back in the day’ they can see the things that spoke to you the most, and not just whatever your fleeting thought for that day was – along with the cruft & things other people thought you wanted to see.

I am sure there are tons of different ways to SEO yourself. I would even venture to say that it’s like a website – it isn’t something you can ever stop doing. Especially because the rules of the Internet are changing by the hour right now.

If you have more ideas about how to SEO yourself, I’d love to here from you. Shoot me a message, and let’s chat about it.

Update 2. Making Progress.

Overview

Well not a massive amount to report this time. Haven’t acquired any new projects lately, so that’s good! This is what the yard currently looks like:

2015-09-01 17.11.54 2015-09-01 17.12.09 2015-09-01 17.12.24

Monster Truck

I worked on the truck some. I ended up buying the rear main seal and replacing it. Replacing the seal part wasn’t so bad, just a two-piece seal that slides in around the crank. But what totally sucked, was getting the oil pan off and back on. I’m not sure if it’s the fact that it isn’t a stock engine, or it’s an oversized oil pan or what. But taking the oil pan off took probably 5 hours by the time I had the bolts out, and the cross bar on the sub-frame moved. I actually ended up breaking one of my ratchets that I was using to rotate the crank to get a little more room pulling the pan off! Whoops!

I think it took 3 hours to get the stupid pan back on, because I got all the bolts except the front two in, and the front two just would NOT go in. The pan was too far back, and even with loosening all the bolts, I couldn’t get the pan to go forward. So I ended up taking it all back off again, and trying again. I started with the front 2 bolts this time, and after quite a bit of finagling with the gasket, I got it all back on there! I did take the time to clean all the bolts, pan and everything else while it was off, so that’s good. (Actually, Heather cleaned the pan for me 😛 apparently that sucked haha).

Pretty much what’s left on the truck is to re-do the fuel line, make an exhaust, and wire in the electric fuel pump to a switch inside the cab. Once I get those couple little things done, I am going to take it to a local shop to get it dyno’d and tuned. Should be about $250, so I’m pretty excited to do that and see how much power it puts down.

Evo

Aside from the truck, I worked on the Evo some more. I got the rear end all back together and ready to put in the car. Got the frame fixed and I also ordered the EGR delete. Installed the EGR delete, which made the intake manifold look WAY better. It took off pretty much all of the vacuum lines on the car. I still need to remove the charcoal canister that’s under the car near the gas tank, but I’m planning on doing that when I put the rear end back in.

2015-08-31 18.58.43 2015-09-01 17.32.03 2015-09-03 23.22.50

I also got the wiring harness all crimped up and done. I just need to get the battery relocation kit to the trunk sorted out, and switch out the positive terminal at the fuse box. I also need to figure out a shut-off switch for racing. I’m not exactly sure how I want to do that, but basically I need the switch to shut off the battery and the alternator, but I’m trying to figure out how to do that without running a bunch of extra wire up and down the car from the trunk to the engine bay. I still need to re-loom the whole harness, but I am waiting to get the engine in the car, and figure out exactly what plugs need to go in the engine bay, so that I can actually group the correct plugs and everything together. Just to make it look a little bit better.

2015-09-01 17.14.23 2015-09-01 17.14.35 2015-09-01 17.14.39 2015-09-01 17.14.50

Aside from that, I got the head put on the block, got the intake manifold bolted up, cleaned out all the oil ports, timed it and did a compression test. The compression was right around 160psi across the board, so I am pretty stoked about that. It’ll go up some more once the rings have set, but at least this time the motor had enough compression to run! This is up from, 90, 110, 0, 0 starting from cylinder 1, going to 4. I guess the machine shop didn’t cut the valve stems, so the valves weren’t fully closing. At least I didn’t bend any valves!

I just gotta finish putting everything on the motor, and put it in. I don’t think it’ll take super long. Just trying to tidy up a few little things first. I also ordered another intake cam gear, and an intake cam angle sensor. The sensor broke the last time we pulled the motor (oops). I ordered the cam gear, because you can’t buy an adjustable one, but you can get it machined so you can adjust the stock one. However, I don’t want to wait for all this to be done, before I can put the car back together. I also ended up buying the intake cam with the gear. I got the sensor, cam gear & cam for $100 from someone on the Evo forums! So that’s awesome, considering that sensor alone is $100 new.

Genesis

A little less work, and more fun, I’ve been taking the Genesis out to the local drag strip. I’ve taken it out about three or four times. The best time I ran was a high 15s 1/4 mile, at about 90mph (I keep losing the time slips, so I don’t remember). This last night out though I got a lot more seat time. The weather and track conditions weren’t as good, and I only ended up running a 16.1, but it felt a lot better and I got to get my shifting and launching more dialed in. There is one more day to take it out this year, so I’m hoping to run a mid 15, which is right about what the fastest people manage to do with these cars stock.

Wrapping Up

My friend William’s car is in one of my bays of the garage right now. That is the car we rebuilt the engine for. The engine is back in, and almost everything is hooked up and ready to go. It should be out of my garage soon, which is super exciting! That’s about all the updates for now. Hoping to get the Evo engine in and get started on the body work here in the next few weeks. But I’ll update again in a few weeks when I have some more to tell!

 

P.S. Sorry there weren’t as many pictures this time. I’m not good at taking those. I’ll try and take some more and post them with the next update!

Swooping Updates! Overview Of The Cars.

So I figure I’ll give this a try. My goal is simply to share what I am doing at the moment, what I’ve finished and upcoming goals. Primarily for friends & family, just so they can see what I’m up to. Updates are unlikely to be regular, but I’ll try and at least get one update a month, and I’ll try and keep them vehicle specific – but that isn’t always going to work. Like this update, it’s going to be a little bit of all of them!

I guess I’ll start off with just an overview of everything. I’ve been working on the Evo, currently fixing the wiring harness and rebuilding the rear end. More on why I’m doing these in a bit. Aside from that I helped my friend William rebuild the bottom end for his Eclipse that he’s trying to sell. Hoping that car will be done and out of my yard within the next couple of weeks!

2015-08-27 21.01.30 2015-08-27 21.01.14 2015-08-27 21.01.04 2015-08-27 21.00.54 2015-08-27 21.00.41 2015-08-27 20.59.35

2015-08-27 20.59.24

I have been trying to clean up and get rid of extra stuff, and projects that I’ve had forever but never got around to finishing. So, the biggest one of these was my red RX-7. I originally stopped driving it when I bought the Evo, and it just kind of sat since then. I tried selling it to a friend, but that didn’t work out, so I ended up with it again and the storage unit full of parts for it. My friend Kyle and I took the carburetor off to rebuilt it, but through a series of events, it never quite made it back on the car. So it didn’t run for like 1.5 years or something.

IMG_20130831_163912_352 IMG_20130831_163819_701

So, I decided to get rid of it, and all the parts laying around. I ended up trading it for a 2000 Yamaha YZ-426 because I figured it would be way easier to sell a running dirt bike than an non-running RX-7 (a car hardly anyone wants in the first place).

00O0O_aBIsakIkVMb_600x450

Buuuuuuttttttt, since I’m me, and things like this are a regular occurence in my life, I ended up trading the Yamaha for a running RX-7 of the same generation, AND a sand rail. The RX-7 is completely gutted, with a rollcage. More pictures of that to come when I started diving into it.

2015-08-30 11.19.49

The sand rail doesn’t have an engine, but it came with a Honda B18 and the adapter plate for the transmission. The motor needs a bunch of stuff before it can go in the sandrail, like intake/exhaust manifolds, valve cover, ecu, wiring harness, stuff like that. So, I’m not sure what I’m going to do about an engine for the sand rail yet. I’m thinking I may end up getting a donor car with that same engine, and taking what I need and then selling or scrapping the rest – hopefully recovering most of the cost of buying the car in the first place. So that’s the latest addition to the fleet of cars in my yard.

Aside from cleaning up, and consequently getting MORE cars I’ve also been on the hunt for SAE tools, and shelves for the garage. I ended up finally finding a full set of SAE wrenches from a garage sale for $5, and bought a set of sockets from Sears for like $20. So, I finally have the tools I need to replace the rear main seal on the Monster Truck! I also found a few shelves from Craigslist and garage sales for super cheap, so I finally have somewhere to put the parts to the 10 cars that are currently in my yard!

Another really cool tool I acquired recently was a drill press. I found a very old one at a garage sale for $20. It didn’t work when I bought it, but after a little bit of re-wiring and a new switch it works awesome! The drill head moves forward & backwards, as well as rotates in 360*, while the table moves up & down. So that’s pretty cool.

2015-09-01 17.13.15 2015-09-01 17.13.36

But enough about organization & tools. Back to WHY I’m fixing the wiring harness in the Evo, and rebuilding the rear end. So what happened with the Evo, is my old one (the white one) was a salvage title, because it was vandalized & totalled out by insurance down in Las Vegas (this was before I got it), and my friend that sold me the car said he would trade me that shell (because it has a widebody kit on it), for this shell he just got that has a clean title. Well, just because it has a clean title, doesn’t mean it’s a clean car! So, out came EVERYTHING from the white car, to go into the new (red) one. But, since the white evo had been converted to an Evo 9 (which has more fancy electronics, like an adjustable center differential for the AWD and variable valve timing) from an Evo 8 there was a lot of work to do to get all of that back into this red Evo which is an Evo 8 (didn’t come with all that). So, I was going to splice the harnesses together, and get everything working, but realized the previous owner had already done that (whoo! or so I thought). But, after de-looming the entire harness, I found a ton of splices that were just twisted together and taped up.

2015-08-14 00.16.37 2015-08-14 00.16.30 2015-08-14 00.16.03

 

Which means I have a TON of crimping to do. It is about half done though, and gives me a good chance to relocated the fuse box into the cabin under the dash, and clean up the engine bay a little bit.

The rear end is a much less exciting story. What happened is all of the US Evo 8s and 9s came with the plates for the LSD stacked in the wrong order from the factory (ugghh). They’re supposed to go friction plate, friction disk, friction plate, friction disk, etc. according to the Factory Service Manual. However, they come plate disk disk plate plate, which really just means the rear end doesn’t lock up as quickly. For normal driving, it makes no difference. For racing, when it’s stacked properly, it means you can cause the car to Oversteer when punching the gas pedal – allows you to enter a turn faster, with mild Understeer and the use the gas pedal to cause the back end to go out and get the car to track again. So that’s the story behind why I am rebuilding the rear end.

2015-08-19 21.42.08  2015-08-19 23.19.14

 

That’s really it as far as the latest updates of what I’ve been doing. Coming up is finishing the wiring harness, eliminating the EGR system for the Evo, and digging into the newly acquired RX-7 drift car. In the future I’ll try and keep the updates a little shorter by doing them more frequently!

WePay For Laravel

If you’re like me, then you’re employer always, without fail wants to use the least common, weirdest things within your application. Well, in my case that happened to be WePay. While not a bad choice (and there were logical reasons for using, in their defense… maybe) it apparently had not been done for Laravel. So initially to get it working, I precariously perched the code & prayed nothing went bad (and nothing updated). Well, after a few months it updated. Damn :/

However, lucky for you guys! I made it work with Laravel the way vendor packages are supposed to work. And here’s the link to it on GitHub:

https://github.com/Ryuske/wepay-laravel

The README.md on GitHub should hopefully sufficiently explain how to install it and get it working. It it doesn’t, let me know & I will update it with your suggestions.

Using Yahoo API with Laravel 4

After nearly a full day of researching and fighting with the Yahoo API because I needed access to peoples contacts so they could invite their friends to support the cause they were fundraising for (I work at MoveYourMountain, thus the context ;)). Anyways, I finally figured it out and thought I would share for those of you who are trying to interpret Yahoos [literally wrong] API.

As I mention in the code, it is pretty straight forward to convert this class to work with vanilla PHP instead of using Laravel.

Here is a link the the Github Gist, if you’d rather view it there.
https://gist.github.com/Ryuske/882651e7bde19afc8feb