Measure twice, and cut once

So here's the situation, I made some changes to the engine that run "Memories Documented" (if you still don't know that the name of this blog is Memories documented, this is a chance for you to know)

Anyway, I made a change to have an enhanced feature seldom found in blogging software, at least I don't know if any does that. But the end result were not expected. It worked as planned collecting all sort of date but what I failed to calculate is the amount of data this is going to generate. Especially when I had episodes of increased traffic reaching almost 500 concurrent sessions. Which is a lot by any standards

During my joy with the fact that I'm collecting this information, I decided (design decision) to filter on the output. As in, collect everything and later on filter the results. And what a grave mistake that was! I totally underestimated the amount of traffic and how much data that will generate.

Few days later I started noticing huge delays on my blog, and excessive system resource utilization. It was time for me to start digging deeper to see what's going on and to my surprise. I found those buckets that I have set to collect date containing Megabytes upon megabytes of information that is being retrieved with every single time the page is loaded. A complete waste of resources

Not only that, in my haste I didn't design properly to accommodate for future modifications. In other words, at some point I'm going to have to restart all over. What a disappointment, this is a text book rookie mistake and I did it too!

Anyway, I fixed the current situation, but I will need to make some changes in the future if I want to continue to enhance that feature. I will eventually do it ... sometime in the future ... mean while... well I'll have to live with half of my goal!