Focus Your Resources
I was recently party to an office meeting where our manager revealed a couple of news items. First was one of our fellow developers is "no longer with us." Hurray for euphemisms. Not that I can blame our manager. I probably would have said something more direct and been vehemently hated for the comment. But this post isn't about our fallen comrade, but instead about the second piece of news we learned. How they were going to fill the position.
Our application is crap. Let me just say this up front. I do what I can when I can, but certain things(what a strangely placed pronoun) prevent the job from getting done correctly. That's for another post. There are two major design flaws to our system.
One of the major design flaws of our application is the use of Aggregate Entities. Aggregate entities are used on our project to gather as much information upfront as possible. That way when you need to process information later in another method, you can save a trip or two to the database because you already have what you need. This is a great idea...in theory. Now I still haven't completely found away around this mess, but one thing is clear, retrieving what you need on demand is more efficient than loading a bunch of crap you're never going to use. Often what happens (keep in mind this is a bunch of devs under unreasonable timelines), one person will create an aggregate entity for a person. This AggEntity will have the Person record, Email Record, Address Record, etc. Each of these records account for the acquisition of one database call (SELECT Stuff FROM Email Where Person). Then someone else will come along and create a Provider entity, which will include the Person aggregate plus whatever information is needed for the Provider. This goes on and on until you hit the Facade level just before the UI. Now you have a ProcessAggregateEntity which includes Page Entities each with its own Page entities which subsequently includes entities like the aforementioned Provider Aggregate Entity. Ugh, it hurts my head even trying to explain it. I didn't build the framework I just work within it (and sometimes try to fix it).
The second major design flaw that we have is the use of a generic object table. This table was designed to decrease the scope of tables that would be needed for the database. The object table, which keeps a record ID for each object in the system, is used in combination with a Relationship table. The Relationship table keeps track of all the relationships between the objects. A quick example: An application (an object) has a bill (an object) which is marked by a Application_Bill relationship. So instead of having an Application_Bill table to keep these relationships between the Application and bill tables we add it to our generic Relationship table and now have a generic relationship between two objects which point back to their respective Application of Bill table. Now when a new relationship is need (eg. Application_Person relationship) is added, a new table isn't created. Instead a generic relationship is created joining the tables. We have tons of these types of relationships in the system and at the time it seemed like a great reuse of table space. In hindsight...WHAT A FUCKING DISASTER.
These two issue add up to one very large problem. A MEGA FUCKTON of database calls. This has our quad-core 64 bit Sql 2005 Server pegged. Almost 100% utilization. Our application serves multiple jurisdictions across the state making performance an issue. Even still, this isn't a stock trading application. The program should be super responsive. It is the opposite. With all the aggregate entities running amok in conjunction with all our relationships being in the same table, the system is making millions of calls trying to form these aggregates. Often the program makes these calls to do something trivial like "Is this Application eligible?" This creates hundreds of database queries to build the aggregates some of them hitting over saturated tables. This is the problem: trying to make one call to the database can result in hundreds of calls.
Bringing you back to my original point...what is happening with the old dev spot? The decision was made to bring in a consultant to make the database server more efficient. WTF! I understand that this decision was made because management has a fire lit under their ass to MAKE THINGS HAPPEN. But what is the point of spinning your wheels for a non-solution? The actual goal that my decision makers are trying to accomplish with this move is to show to the Executives that they're responsive. "See...we're doing something about it." It's a PR thing, which I despise (however PR is a necessary evil as I've learned). What really burns me is that money is being burned up just to try and prove we're doing something about the problem when we ARE NOT. Despite whatever performance gains are realized by this new individual, the system will still crawl because it is designed poorly for scalability. We might as well be flushing the cash down the drain, because the cost effectiveness of such a move is atrocious.
The real solution consists of taking the bull by its horns. Grab up a dev and set him to task on refactoring a slow part of the system. DO NOT HIRE A NEW DEV (Rule #3). Tell the dev to do what it takes to realize this performance gains. It may take a couple months to get it right, but when all is said and done you will have a REAL performance gain. No amount of configuration can help a system that is poorly designed. Target a module and fire. Even if it takes you 10 magazines.
Develop good software. Realize when your code is crap and aim to change it. Only so many band aids can be applied to a machete wound. You will soon have to remove the band aids, disinfect the wound and sew it up in order to repair the damage properly. I think that the executives would rather have a real win versus wheel spinning dressed up as a win. Focus your resources like a laser and pull the trigger.