My apologies for the delay in giving everyone a thorough update, but I was up and down almost 36 hours addressing the outage, and I needed some sleep. I’m awake now, but at 73 with a triple by-pass, I’m not sure how much longer I can (or want) to do this. It’s been a challenge, but for the most part we are up and running with one major board function that is unfixable, and a whole bunch of minor ones that are either fixed now, or not worth fixing since hardly anyone ever used them anyway.
Background
This all started after we were informed that after more than 15 years of operating without a major outage, all our software modules were at “end of life” and potentially vulnerable to external exploits by unsavoury characters. I know that feeling, because right now I feel my “best before date” has expired.
These software elements included the base Unix operating system, PHP, SQL Database and what you all see here as the vBulletin version 4 web site. We wanted to protect all of our 35,000+ members, so we spent a lot of money proceeding with what was supposed to be a planned 4 hour set of software updates. Of course, no plan survives first contact with the enemy. I should have remembered that from my military service.
Execution
In simple terms, the plan unraveled in real time and the update failed completely. We couldn’t make anything work well moving forward, so we decided make a tactical withdrawal to where we started. That’s when we discovered that some of our previous software modules were no longer available. Trapped in no-man’s land, we had our in-house tech support guy do the best he could. He eventually managed to “jerry rig” elements of the different modules and finish what he could at his level. He then recommended we get experienced developer support for vBulletin. Following his advice, we hired a very talented external vBulletin expert, to patch the old vBulletin 4 software for compatibility and arrive eventually to what you see now.
Where did the post attached pics go?
In the middle of this mess, we had also been paying another site to backup daily, the critical database files, which we assumed included the pic attachment directory. Well, the moral of that is never assume anything. Apparently the directory shows in the backup listing as being backup up, but it was EMPTY. We’re still waiting to hear from the company we paid to do the daily backups, as to why that specific directory was the ONLY one never backed up.
The effect on members is that ANY picture they attached to a thread/post using the site’s internal upload capability is permanently gone. As the short scrolling banner at the top the main home screen of the site says, If your post is important, re-upload and attach the missing pic(s) in the SAME thread. Note: All Knowledge Library and other images imported by external referencing using web links are fine. It was noted that FREE Registered Members are not allowed to EDIT their original posts and threads after 1 hour. Therefore, I have removed this restriction temporarily to allow that user group to re-upload their pics (if they still have them) to older posts and replace the missing attachments. I will leave this in place for 60 days so FREE Registered Members have the opportunity to update their old posts..
I’m having intermittent problems uploading pics and attaching them to my posts/threads?
I am too! This second issue which is still in-play is UPLOADING and attaching pics to posts/threads, which only affects some members, and not others. When the selected pic from you local computer finishes uploading, the resulting uploaded pic in the very bottom box seems to be a blank white box (empty?) We’ve been advised to give the automated software more time to clean out references to non-existent pics, and see if the issue resolves itself. Please report problems you are having to me badger@milsurps.com, supplying as much detail as possible so we can troubleshoot this further. Make sure you include your site USERNAME and any screen captures that might help us identify why only!
Summary
I take full responsibility for this fiasco, making no excuses. I should have ensured the attachment directory was indeed actually backing up and not just appearing in the backup listing. At a minimum, I should have ensured that directory had a redundant backup mechanism elsewhere. Apologizing to all of you is no solace, but I sincerely offer it and will take steps to ensure that this never happens again.
Finally, I want to thank Paul M. and Darren T., for their awesome help in getting us back to somewhere near normalcy. Without their valuable technology knowledge, we’d still be down and out in the cold.Information
Warning: This is a relatively older thread
This discussion is older than 360 days. Some information contained in it may no longer be current.