JournalSpace Drama: All Data Lost Without Backup, Company Deadpooled
by Robin Wauters on January 3, 2009

Blogging platform JournalSpace (which I’d never heard of to date) has ceased to be, following a wipe-out of the main database for which there was no back-up in place. According to the JournalSpace blog, the database was overwritten as a result of a malicious act from a disgruntled ex-employee.

It was the guy handling the IT (and, yes, the same guy who I caught stealing from the company, and who did a slash-and-burn on some servers on his way out) who made the choice to rely on RAID as the only backup mechanism for the SQL server. He had set up automated backups for the HTTP server which contains the PHP code, but, inscrutibly, had no backup system in place for the SQL data. The ironic thing here is that one of his hobbies was telling everybody how smart he was.

More information is available here.

The company set up a Twitter account which it used for updates, and reading the messages in reverse chronological order is telling enough. If your blog was hosted on JournalSpace, you can visit this helpful blog post with instructions on how to save most of your content and comments using Google Cache.

JournalSpace had apparently been around for 6 years, and will now be releasing its source code to the open source community, and possibly sell off the domain name and trademarks.

By the way, when’s the last time you backed up all the data on your computer?

(Via Slashdot)

Advertisement

Responses

Comments rss icon

  • The only part I don’t get about their story is that if you check the community forums they link to on their site, people are getting backups of their data… How are they getting these backups if they lost everything and it wasn’t recoverable?

    • There are instructions on how to recover some of it using Google’s cache, perhaps that’s how?

    • This is why users must be able to own and control their own private information.

      Just as we need regulation to ensure that money we have in the bank is protected, we need some regulation to make sure that personal privacy and data is protected.

      • Server in my basement, rsync’d with some rented server space, the rest of the net can burnout. Currently data protection is on whoever owns the server, and if it’s not yourself you will always be at there mercy. Regulation of data at a server doesn’t mean you will have access to it. I suppose we can force regulate people to keep bandwidth in place? Or that the power or grid won’t get interrupted? Or that they have proper network security for when the russians ddos the server(s) for weeks? And that you can’t sell the server? I liked it when companies just buy up pirating servers to grab all the info on it for lawsuit purposes.

      • Simple solution to this is simply having multiple back-ups with different hosting providers and rotate them sequentially. Throw-in a few on your own computer as well for good measure. Regulations for doing something that should be done anyways doesn’t solve the problem… only creates more paperwork.

        Aside from that, I think the CEO of this company only has themselves to blame for the problem. Why? Because he should have realized that without proper backup systems in place (which he should have verified at least once a week minimum), he was always one hick-up from loosing not only his business but all the work from his clients as well.

        IT guy is a scape-goat… ignorance is not an excuse.

        Jon
        http://buzvia.com – Where’s Your Traffic Going?

      • “This is why users must be able to own and control their own private information.”

        Exactly. Spend $10, register a domain. Find a reputable hosting company (do your research) and pay the $100-$200 for a shared account. Install WordPress. The WP control panel has an “export” button, click that and you can download all your posts and comments as one big XML file. That’s it!

        Never, never depend on your host. Something will go wrong, eventually. Even that data-center island guarded with machine guns, even that thing burned up in a fire. Floods, robberies, it’s always something.

    • They point out that the guy who caused this is dumb. But they were screwed by him.

      Does that make them even dumber?

      I think so.

  • It will be good to get the name of the “guy handling the IT”, just to recognize the bastard.

    • He mentioned http://www.lagomorphics.com/

      If it was them, I think it is safe to say that they are deadpooled too.

    • I’m calling shenanigans on the “malicious IT guy” angle. The only thing mentioned about was the choice of RAID with nothing about any bad acts. “Didn’t setup db backups” is not a malicious act.

      Be careful you don’t fall for the line that only serves to save the business-guys’ reputations. Someone approved the hardware purchases, (apparently) didn’t ask the right questions about business continuity, and is now for some reason calling someone “the guy handling the IT” rather than anything describing the role or position he had in the company. That is, “what, did you just hand over all the keys and money without question?”

      That after a few days and much attention there is no arrest or lawsuit says a lot. And yes, if you let someone kill your company you take steps like this to show you’re responsible. Think about it this way: what did the company do to piss this guy off so badly? Don’t be so ready to call everyone but the unnamed “IT guy” innocent.

      • Yeah… I’m inclined to believe the same thing as @EH above. Something stinks in their explanation.

      • I’m with EH. Lets see. I have a business that is at least operating, perhps even at a minor profit. I have an ‘IT guy” who comes in and screws me over. Stealing from me (who I dont have arrested?) and stupidly I just ignore the fact his hands have been all over my business. Fast forward a bit, and my data is wiped out and so is my business which I do believe is a FELONY and I am going to just fold up and take my toys and go home? Sounds like a movie script to me. Here is a better potential story.

        I’m trucking along in my business, staying afloat but barely. Along comes a recession and things get scary. Then I find an “IT guy” to blame for the whole thing and close up shop before the company bleeds too much due to rising overhead. I then bid my users farewell and auction off the only tangible piece of property I own, the domain name.

        My script sounds like it would sell more popcorn.

      • It could be both a malicious IT guy’s fault and his own.

        I was running a site a few years back hosted with “The Planet” (when it was ev1). I was paying an “IT guy” in Belgium to take care of my servers and make backups since it was a lot cheaper than hiring someone locally. One day, about a year after we launched, the site completely vanished. I called my hosting company, and they said that they had wiped all the data on my server because it was running an IRC file-sharing network and I had been amply warned that I would have to cease that activity or be wiped. There wasn’t much I could do since the email they had on file for the account administrator was this same guy in Belgium, and he had simply not forwarded the warning emails to me. When I considered my options, I didn’t have the means or the knowledge about how to sue someone internationally and I couldn’t sue the hosting company since they had technically attempted to warn me. So it was my fault for not doing my due diligence in watching our servers or monitoring my “IT guy,” but you could also blame the IT guy for being stupid and running a file-sharing network, and the hosting company for not calling me before taking such a drastic action.

    • did this happened at the new year’s eve? http://groups.im/

  • hmm that is an epic fail. Inside jobs tend to be by far the worst sort of security attacks on companies IMO.

    I operate a similar style site Path of a Hero, but it targets the World of Warcraft community. I have been around for only 2 months but I can assure you that I make sure that there are adequate backup procedures in place :S

  • This could be a candidate for a Daily WTF post. So they backup the PHP source…which probably doesn’t change too often, but they don’t backup the users’ data, which changes daily? Seriously, what were they thinking?

    • Even if they had a back up, if the guy had access to production data, he would most like had access to the back up, so he could have deleted that as well. I really do not know how you can avoid an inside job, maybe haveing law that will deter this act, like a lot of years in jail for that bastard.

      • Luca: Sure you can safeguard the data. Set up an off-site backup destination (with multiple generations of backups) and give your staff write rights only, but none of the delete rights. This isn’t expensive or complicated. The management there is just as much to blame for not establishing proper enforceable in-house policies.

  • Here’s the proper way to back up your computer:
    http://www.jwz....oc/backups.html

  • Awful! Another way to retrieve your text content (well, some of it probably) is from your RSS feed if you subscribe to your own feed. However, that would probbaly only give you your last 200 posts or so.

    Another way is if you subscribed to your own content via email. if you kept all those emails, you’ll have that.

    These are all workarounds, so to speak, no substitute for actual database backups.

    Maybe the Wayback Machine as a last resort.

  • Backups never work for me. Last time I burned most of my data onto 10 DVDs and kept them safe for 6 months. After formatting my HD I turned to them, only to find out that my PC didn’t recognise them. I lost most data and the mp3 files were scratchy.
    :(

  • Some IT guys definitely have too much power.

  • i had one of the oldest journals on journalspace. i am really upset about losing about 6 years of writing, and my layouts which i made. it was bad enough when they lost years of comments. this is far worse. i am pretty sure i archived most of everything up til about a year ago on my external hd. i’m actually a lot sadder about this than i thought i would be. to all of my js friends, please find me:

    http://lifestre...fm/tinythoughts
    http://twitter....om/tinythoughts
    http://www.new....hp?id=818435715

    • Louis-Eric Simard - January 3rd, 2009 at 10:52 am PST

      That truly sucks. I think the article above should be updated with what you just brought up: that they already had a data-loss incident before; that makes blaming the “web guy” a lot less credible if management didn’t wake up the first time around.

    • If you value your work so much, you shouldn’t be using something that’s free and expect not to lose it.

      At the end of the day, piss all you want. It’s your damn fault for leeching off a free service and expect it to continue to provide for you.

      The day of FREE is over!

      • dumb hal, it wasn’t free. i was a paying customer for most of the time i was on there, until the first big data loss. after that, i did not get a pro account anymore and also began to write less on there.
        as for losing my stuff, which i did value, as i said, i did back it up myself after that first data loss. however, i liked it where and how it was, accessible online to me and anyone else. we are talking about almost 6 years of content. that is not some small thing. even if you’re dumb, you should be able to understand that.
        btw, i work in this industry myself, and i am pretty sure the days of free are not over. but all the best to you on being rude anonymously to others online.

        Louis-Eric Simard, I don’t believe their story. I think there is more to it. They’ve had problems before and they always lay it out like their users are technically stupid and willing to accept any dumb answer given to them. What happened really was a great loss for many users, who had been there for years, a community of really great people. The greatest loss is all of the time, life, love, and community each of those users put into journalspace, where it was all documented and washed away like sandcastles on the beach. I might be upset for my own loss, but not nearly as sad as I am for many of my friends there. I think journalspace owes them more than a lame excuse and an empty sorry.

      • While your name is clearly on target, I want to suggest that you start going with “Rude A**hole”. So you just couldn’t resist the opportunity to make light of other people’s misfortune, could you? I guess you’re too “dumb” to have noticed that life has a way of serving up a double-helping to thick-headed, soulless people like you. Enjoy your karma!

    • I know what you mean. I had my JS account for over six years and my readers paid for my account. I feel sorry for them since money is coming out of their pockets to read something that doesn’t even exists anymore. I didn’t care for the lost written content as much since it was not my primary journal account but it was the best looking one in my collective as I’ve spent a bit of time customizing it and now it’s gone.

      Don’t expect me to start a new account with the current JournalSpace service. I would rather reactivate one of my deleted journal accounts elsewhere than to go back to JS. A huge shame for the people in my f-list… I think I remember only a small handful of readers and I don’t know if they use any other journal service.

  • lol, gotta love an internet company that has ‘a guy handling IT’. As if the IT side of things is an afterthought-which apparently it was in this case.

  • iampriteshdesai, that’s why I only use ’standards’ file formats as backups (tar + gzip, tar + bzip2..), so you’ll always be able to recover, even by hand if needed. + partition dumps. Incremental backups. (Auto?)-test of backup files. No fat-32 nor NTFS filesystems as external backup support (because of some 4Gb or other limitations/surprises..).
    Yeah, that’s theory, we all know that we can appreciate our backup’s quality when a restore is needed.

  • I don’t see how this is a RAID vs. backup issue. It’s sabotage, and the “IT guy” could just as well have also messed up the backup, even if they had one.

    • Very, very good point sir. He woulda just pissed the backups up, if there were any. But they fired his ass a couple months ago… the idiot who ran the joint could have done something…

    • What story are you reading that would confirm that it was “sabotage?”

      • On the journalspace blog, he goes on to say that the dude he caught stealing from him did something that wiped out the data…

      • Yet there’s nothing to say what “sabotage” could have happened. Lots of finger-pointing but no technical words at all. From what we all know, the only thing “malicious” was that there were no db backups being run, which is only “malicious” or “sabotage” if it was some mind-fuck by the IT guy to convince the CxO that no backups were necessary.

  • Israel rocks!! I just wanted to throw that out there. WE OWN SILICON VALLEY!

  • I could image how i would be disappointed.

  • hmm ex-employee was smart enough to destroy data and yet CEO wants pat on back??

    Story of Epic Fail CEO Story..the TechCrunch Chronicles..

  • re: backups not recognized

    That sounds like a program was missing or reconfigured on a personal machine.

    I try to keep seperate backups of drivers, programs and data files, and include text files with notes on current configuration so that I don’t waste time trying to figure out what I did and how I did it if I run into problems.

    I use an external hard drive as a primary, with an additional set on jump drives or SD cards. Anything important gets backed up immediately, using “Save As,” to the machine, external drive and jump/SDs.

    And yes, once a week I have to do light maintenance, in the form of compressing, sorting and verifying, as well as a monthly, more in-depth version that includes assigning new file names for long-term retention. Once a year I convert backed-up data to newer formats and either delete or convert to text anything that has lost relevancy.

    This has saved me on more than one occasion, which is why it is such a neurotic, paranoid approach, but I can open files originally created on a Commodore 64 and the dinosaur XT portable (not that they have any value beyond sentiment).

  • Even if they had backups, unless the staff kept them offsite, the IT guy would have wiped them out too. This was a personnel problem, not a technical problem…

    • ridiculous thread. depending on the level of the IT guy, if it’s an inside/malicious job, then it’s a tough situation. as well, irrespective of the level of the IT guy, they probably know more about bcp/disaster recovery than many CEOs. If there’s a VP of engineering, cio, cto class guy, then probably ok to share the blame with “management” … which is not necessarily the same as saying it’s the CEOs responsibility. Though, to be clear, at the end of the day, responsibility always ladders up to the ceo — if the company failed because of this issue, he should be toast as a solid ceo candidate.

  • Wow, that’s strange. Think Blogger loses its database. :D The sky really falls down on all blogger users. :)

  • Scary. ANY site of any kind that stores server side data MUST have backups, and backups of those backups. This is a basic rule of business on the internet. Amazing.

  • sorry.. but i got to laugh…

    and this is a site that guy k saus is important for startups… no wonder the vally crowd is fucked!

    to begin, raid is not a backup policy. it’s a redundancy issue for the data. if you hire someone or someone tells you raid is backup, run. backup is backup, pure and simple.

    a good backup is also more than a simple rsync script to copy the data from a principle to a secondary drive. while this could work, it might not be sufficient.

    a good backup policy for a work environment is far more than a home system.

    there are plenty of articles on the ‘net about backup systems/policies/strategies…

    good luck!!!

    peace

    • Louis-Eric Simard - January 3rd, 2009 at 10:56 am PST

      Agreed; an on-site backup is completely useless if fire, flood (think about the sprinklers turning on if you are set up that way), or theft occurs. The on-site backup should be a speed-recovery method only, but not the definitive data safekeeping line of defense.

  • This is why I’ve always thought that every company needs an outside technology audit from time to time. As a business owner, even in the tech arena, it’s unlikely that you’ll have the tech know how or time to sort through what your tech person or team is doing and be able to make determinations regarding it’s quality and safety and security.

  • Too much power on the hands of one single person?

  • First thing I would have done is hired an outside firm to come in and see if they could find the lost data. Even after formatting a hard drive you can still get old data off of it. Sounds like they just used this as an excuse to let the company fail.

    • what part of “DriveSavers called today to inform me that the data was unrecoverable.” did you not read?

      the whole website? Or just that part?

  • Have to agree with Nick. This was a personnel problem not a technical one. That said, prime example of why you need a verified backup strategy in place before disaster strikes. Fail JournalSpace.

  • Well, they hired DriveSavers according to the article. Did you click on the link?

  • Wow…no backup plan and no offsite storage? Isn’t this disaster recovery 101?

    Hopefully they sue the crap out of the “IT Guy”…sice he knew what he was doing when he was destroying the database. I wonder if he took a “secret backup” of the database?

  • Pretty dumb move firing an IT guy like that who has all the keys.

    How about backing up everything and changing all passwords & logins BEFORE firing the guy handling your IT.

    • This is prob. how it went:
      IT: ’sir, we need backups of data, I’m telling you for the last time’
      CEO: ‘oh yeah? what’s that going to cost me?’
      IT: ‘$xxxxxxxx’
      CEO: ‘no can’t do’
      IT: ’so what if we lose all data?’
      CEO: ‘don’t be silly, how could that happen’
      IT: ‘rain, flood, tsunami, earth quake, giant magnets, coffee, burglary, shall I continue?’
      CEO: ‘I’ll take my chances’
      IT: ‘if you do this I will have to get a big raise for taking such a risk. It’s my reputation on the line’
      CEO: ‘WTF will you just get back to work’
      IT: ‘not until you answer me, protect my reputation.’
      CEO: ‘that’s it, moron. I can get you replaced by tomorrow. You’re fired.’
      IT: ‘ehm…’
      CEO: ‘get out’

      and that’s how it ended.

  • hadise dumtektek song!

    You can use the poll to say your view about this great song.

    http://dumtektek.blogspot.com

  • I am wondering why JournalSpace’s non-IT management placed the most important aspect of their business with “an IT guy” and did not require additional consulting to verify system stability and security.

    My employer’s policy is to disable an employee’s user ID and password as soon as they step into the meeting in which they are to be terminated, and are not allowed to touch the system again. Employees with high-level system access are ALWAYS escorted from the meeting to gather their personal possessions and to the door.

    I am wondering why JournalSpace did not have a similar policy in effect.

    The next question would be what type of legal action they will pursue against the IT guy, as his actions (as well as their naivety) destroyed the business, depending on the provisions of his contract.

    Finally, I wonder if their abandonment of the entire JournalSpace experience means that they were going under anyway, and would rather be remembered as a black hat hacking victim than an example of poor management and casualty of the economy.

    • Its a lot like trusting your auto mechanic. Unless you have him explain every step he takes and double check his work every bolt along the way, you never know what evil they could slip in. A good programmmer/IT person who is at the top of the IT ladder in a company could potentially put any number of back doors in. Just look what that Administrator did to the entire City of San Francisco. Some IT people who are really nasty will even install ‘dead man’ code that works much like the dead mans throttle in a train. If you release the system for too long it starts running code to do whatever you wish. Could lock the system down or wipe data, or open a VPN connection for you providing a back door……

  • Something smells fishy. Why don’t they have this “IT guy” on a stake? Because if he actually did this, it would be illegal.

    The company is closing up with no questions asked… fishy, fishy, stinking fish!

  • How did they let this happen? This problem should have been nipped in the bud.

    IT guy: I was thinking of relying entirely on RAID for backups.
    IT guy’s boss: No.

  • Are they letting that guy get away with it??

  • First and foremost, I agree that they should of had and on-site and off-site backup plan in place and this would’ve never happened. For all of you saying that the mysterious “IT” guy would’ve had access to that data as well, that’s just speculation. Just because he was allegedly fired and allegedly caused this disaster, doesn’t mean he still had access.

    For instance, he could’ve setup a scheduled task, cron job or something of that nature to drop the database upon his employee record getting deactivated or deleted. Or perhaps he programmed himself a backdoor webapge with unrestricted access to database where he could fire off commands at will. There are many ways a shady IT employee can get initiate an attack at an employer and it’s difficult to build a line of defense against all of them.

    The solution, obviously, is off-site backups. You get the data back in place and then you start searching for the backdoor that was used.

  • I had a JournalSpace. I used to post to it years ago. I guess it’s gone now, too. It wasn’t the greatest site…years behind on design and features – pretty old-school. But it pains me to think what people who journaled there for a long time lost. I use a similar service (LiveJournal) now and I worry every day about the same thing happening to my two blogs. My answer to it is to back up to another blogging site (most of my AOL stuff is on Wordpress for that reason, but I won’t disclose the URL until I need to) on the premise that both sites won’t suffer data losses simultaneously. I can’t think of a better way to back up, since my computer is not always with me nor am I always with it – especially when I travel; with back-up to another blogging site, I just have to distribute the URL via Twitter to get everything going again.

  • this type of situation may have been acceptable 10 years ago; but today, huh uh….it’s not just the disgruntled employee’s fault

  • Well I have my website on http://www.nosle.com.
    So far it is ok. :)

  • far too convent. Think about this scenario – site is loosing traction – so hell – go out with a bang. Make up this story – and get slashdotted/techcrunched/every other blog’d and and drive up your valuation/sympathy to a buyer. But – now is not the time to have done it – economy sucks, hard to find a buyer.

    Best of Luck JS.

  • The first thing everyone should know here is that the public is likely not getting the true/whole story. Every year, many companies, including Fortune 500 companies, lose critical data through disaster, theft, malice, HW/SW failure or user error. The dirty secret is that if they can cover it up, they will. Why? Because such egregious losses will affect stock prices, people’s jobs, and numerous other monetized aspects of a company’s existence.

    The only reason we are even hearing about it in this instance is because the company screwed up so badly that they could not recover. Frequently what happens is hours or days of unexplained downtime or data loss which is either never explained or explained away on some unlikely error. Even Amazon (who sells “cloud” computing and storage to other companies) had a very large outage recently, though I do not recall if it was data related or just compute capacity loss. No one is immune – statistically, it’s just a question of time and severity.

    Whatever the cause of JournalSpace’s catastrophic data loss one thing is a certainty: it was avoidable. Any company that wants to survive a data loss MUST have regular backups of a verifiable nature both onsite and at a geographically remote redundant location – preferably 2. If you’re not certain that your company does, or you’re taking the word of just one person, hire an outside expert to be sure. It’s cheap insurance. Your home computer is no different.

    This kind of failure happens every single day – this is just one of the few that we got to hear about because JournalSpace couldn’t cover it up. What caused it is now irrelevant – it was fatal. Everyone should heed the lesson.

    Mike Voss
    Datarchive

    • Good comment. Also, something that is worth mentioning here, if someone else hasn’t, is the lasting damage to the managers’ reputation and credibility. What a sloppy way to conduct business!

  • wow, that sucks big time

    yes people, back up your stuff…. use sites like ours or other systems to create alternative back-ups, too. The more the better. Cannot stress how important this is. I think a lot of content sites may bite the dust this year due to financial problems, and this may happen more often than anyone would even want to think about….

  • If not true then this is a great PR stunt.
    All the free branding and the interest generated.

  • Quote from JournalSpace:
    “So, after nearly six years, journalspace is no more.”

    6 years running and no backups, I mean c`mon, you were just asking for trouble.

  • Wow. If you thought data backup wasn’t important this should be your wake-up call. If you think it’s too hard Mozy will automate the whole thing and store your data for you. Doesn’t get much easier than that.

  • Much of the discussion here is about how the data was lost, which is good to understand. However, data gets lost for a variety of reasons – whether hard drives that die, sabotage, fire, or a myriad of other scenarios – and having all your data backed up offsite is key.

    I, of course, recommend Backblaze (www.backblaze.com) – but whatever you use, backup your data.

  • Couchsurfing had a similar tale from June 2006 – ironically, with a VERY happy ending!

    More on their data disaster (and its positive impact) at http://blogs.ve...data-disasters/

    Couchsurfing even have a page on it on their website at:
    http://www.couc...cs2phoenix.html

  • I just found the Techcrunch (2006) article relegating Couchsurfing to the deadpool (before they grew from 48K users to 780K users):

    http://www.tech...elf-shuts-down/

    Now there’s an idea for a new Techcrunch story!

  • Did anyone go to the JS site and see what the high bidder on the eBay auction for the domain name gets (along with the domain name)? Incredibly: HOSTING! Yes, with the same guy who self-hosted Journalspace. I kid you not – could you make something THAT crazy up?

  • What is amazing (or not) here is that everyone (except for a few inter-related topics) agrees that you have to back up your data.

    Guess what? That’s not news.

    The other thing is that everyone agrees that this article and the other available information does not describe a complete story. And again, when a company fails, for what ever reason, the likelihood is, you’re not going to get the real reason. You are likely to get a bunch of other reasons that (as pointed out here) are the symptoms of problems, not the problems themselves.

    I’ve spent too many years consulting for too many companies where the managers, directors and C-level folks really believed they had a backup solution. These folks only found out they didn’t have a solution when there was a failure and they couldn’t recovery from the backups. (some survived, others not so much)

    Was this the CEO’s fault? The IT guys (gals) fault? Yes to both. You can always find a way to copy valued data somewhere else (several solutions are mentioned above) so the IT guy has to take some blame. But, the leadership of the company has to take on the blames as well.

    I’ve been the conversation listed above when IT asks for money, and it sounded very close to the actual dialog. That is a resume update moment for the IT folks. Because (right or wrong) if the systems mess up, no matter how many times you have asked for resources to correct it … you, the IT guy, WILL get the blame.

    jam

  • haha. They setup the backup server to copy the main server and when the main server wiped the backup did the same!

  • Funny but scary. The sky really falls down on all blogger users :)

  • “So, after nearly six years, journalspace is no more.” :) tnaks

Leave Comment

Commenting Options

Enter your personal information to the left, or sign in with your Facebook account by clicking the button below.

Alternatively, you can create an avatar that will appear whenever you leave a comment on a Gravatar-enabled blog.

Trackback URL
bugbugbug