While looking at Hacker News today, I came across this article, "I bet you over-engineered your startup", by Swizec Teller. The post goes into the engineering decisions that must be made, primarily between many small, discrete services, and one monolithic service.
Small & Discrete VS Monolithic
For anyone that has had a large project before, the idea of small discrete services or tasks makes sense. If you have a to-do list of many easy to acheive goals, it is easier to move through them and feel like you're making progress. The same can be said of development. I can build 10 discrete services that are incredibly simple, but do their jobs well. Unfortunately, this falls into the trap of being great on paper, but bad in practice. Whenever a change has to be made to one of the services, it is likely that the other services will have to change as well. Additionally, as the system grows, the amount of message passing between services will likely increase. Then the system ends up spending more time talking to itself than actual users.
At the opposite end of the spectrum is the idea of a monolith service. Everything is self contained, creating one large failure point. This also makes development more difficult, as I have to know how the whole system works to do anything.
Teller reaches the conclusion that the services should be simplified by combining related pieces. There is still the separation of tasks, but the tasks are grouped. This is a great idea, and definitely where developers should end up. What worries me is that this wasn't done in the first place.
Don't We Know Better?
Shouldn't developers have the foresight to know that services should be grouped? Shouldn't things be engineered to be maintainable and efficient. For instance, if you are going to pull data in from external sources and store it for use later, shouldn't you have one system that pulls data from all of the services you want? I'm not saying one monolithic piece of code, but a queuing system that can handle different jobs. All of the data jobs are discrete, but run in the same sandbox, so to speak. Then you can build in redundancy, requeue on failure, logging, error notification, etc in one place. This gives you all of the advantages of discrete services without separating your data everywhere. You can run aggregate jobs without asking different services for data. This could be extended to many other use cases. Most are backend data processing, but that is where I have seen most bugs come from.
Much of the code I have seen in projects I have inherited showed a bad design sense. I'm not talking about visual design, though there was some of that, but the design of the services. It's not even differences in coding style, preferred tools, language, or any of that. I saw a job queue that would encounter errors, update the job status, and then archive the job. No notification, not larger logging system, nothing. There was no way to know something went wrong until you started looking for it. I've seen data fetches that just skip over bad results and continue as though valid data was recieved. I've seen web applications that have over 100 dependencies and no documentation to speak of.
Those examples come from large systems. I'm not talking about some site that gets a few hundred visitors a month, I'm talking about sites that get at least tens of thousands in a day. I know all of the excuses that developers can make for these as well. The first that springs to mind is that there wasn't enough time. I've even been there. The difference is, I stopped letting myself use that excuse.
No More Excuses
I think all developers should stop giving excuses. If something can't be done properly in the time given, say so. Don't be a dick about it, just tell the truth. The system cannot be built properly in the time available. Then you have to follow up with alternatives. What changes can be made so the system can be built properly? What can be taken out to give you time to put in proper logging? What can you do now to make sure you can save time on the next project? Build a logging framework you can use and like. Use an existing one. Find a queuing system you like, or roll you own. If you can do these things, you'll only make your job easier.
Inevitably, you'll get pushback from your manager/boss/employer. Be tactful and explain the situation. If they don't buy into it, leave. A good developer should have no problem finding a job right now. I get contacted by recruiters on LinkedIn all the time. AuthenticJobs is putting out new listings every day. Indeed.com has thousands of listings for developers.
Don't be afraid to do good work. Don't be afraid to go somewhere that will let you do good work.