Comments on: Joyent Suffers Major Downtime Due To ZFS Bug

By: Reality

Reality — Tue, 12 May 2009 16:46:26 +0000

More than a year after this posting, it is amusing to see the flame war and religion here. The fact remains, ZFS IS NOT ready for prime time. We had a server crash recently. It was running ZFS and hosting a very large and mission critical Oracle database. It’s not the first time the server had crashed, but it was the first time the server crashed since ZFS had been implemented. During all other crashes, Oracle had been able to complete a full media recovery. Not this time. ZFS had corrupted several data files, all the redo logs and all the control files. We had to rebuild the database from scratch.

ZFS has some good features and will one day be a great storage management solution. But today, anyone running it in an enterpise data data center is playing with fire.

By: warpedvisions.org - » Things best left unsaid

warpedvisions.org - » Things best left unsaid — Wed, 08 Apr 2009 22:26:59 +0000

[...] That’s a big claim to make, especially considering Joyent’s breakup with Twitter and other recent uptime issues. [...]

By: Hosting gone bad: EVERYWHERE!!!

Hosting gone bad: EVERYWHERE!!! — Tue, 22 Jan 2008 04:39:12 +0000

[...] Popular for their reliable Accelerators, they experienced downtime in two of their hosted products during the better half of last week due to a ZFS [...]

By: Slapper

Slapper — Sat, 19 Jan 2008 16:20:13 +0000

Sorry I wanted to say opensource without a commercial support.

By: Slapper

Slapper — Sat, 19 Jan 2008 15:33:33 +0000

1st I would never run mission critical instances/data on open source operating systems.
We ran into a bug with redhat AS4 U1 (and this is not opensource) and its lpfc driver that whenever a registered state change notification is sent from the fiber switch (this is sent even when a zone is created) all the luns assigned to any RH node on U1 would move into read only mode. This caused one of our Oracle databases under heavy load to suffer major corruption.
To just start slamming technology openly usually highlight shortfalls in the business continuity plan for the affected data/customers.

By: Geoff Cheshire

Geoff Cheshire — Thu, 17 Jan 2008 18:57:50 +0000

I’ve been a Joyent (well, TextDrive) customer since January of 2005. While Joyent has had it’s share of bumps, I feel I’ve received a good product at a good value over the long-haul. The community is a great asset, and they provided platforms for doing interesting web stuff.

One statement above that’s misleading is that Joyent was just running a “stale” version of OpenSolaris and got what they deserved. There may be some room for criticism here (I don’t know, I’m not a guru), but people like Ben Rockwood and Mark Mayo are not amateurs. They held back on upgrading the OS for a reason.

[D]oing upgrades on things like thumpers is non-trivial and high-risk, and so we watch the commit logs and bug reports and decide what constitutes a critical problem that demands an upgrade, and what doesn’t. The root cause of this problem isn’t yet fully understood, but I can tell you that we’ve hit several apparently “unique” and “impossible” conditions that definitely have not been reported in any existing bug or commit log.

I really wish it were as simple as “stay on top of known bugs and update accordingly”. Really, I do.

http://www.joye...-update#c008480

By: David Young

David Young — Thu, 17 Jan 2008 01:15:46 +0000

I am the CEO of Joyent. I wanted to make one clarification, because the headline of this post seems to imply that all of Joyent is suffering an outage. The vast majority of Joyent services are up and running without issue. While we haven’t yet restored service for Strongspace and Bingodisk, the ZFS issue are in no way affecting our Rails, PHP, Java, Python, etc. hosting customers. Joyent Accelerators continue to provide excellent on-demand service for Joyent’s thousands of customers including one customer driving nearly 20 million page views a day.

By: Dan

Dan — Wed, 16 Jan 2008 23:03:40 +0000

I’ve been using Joyent servers for a while and I’ve had good experiences, save for growing pains on the shared BSD servers a couple of years ago. I think this is a similar occurence, though it of course stinks for those who are affected. I appreciate the company’s transparency, and would still recommend them to others (once this is sorted out of course). And as for Frank54’s unkind words, I’ve found Solaris to be quite stable and adequate on my medium Joyent accelerator.

By: Kristie Wells

Kristie Wells — Wed, 16 Jan 2008 20:50:46 +0000

Disclaimer – I work for Joyent.

We are keeping our customers up to date on the issues at hand and have posted an update on our corporate blog. http://www.joye...ingodisk-update

By: Jason

Jason — Wed, 16 Jan 2008 17:41:20 +0000

Probably EngineYard now.

By: Jungle Fire

Jungle Fire — Wed, 16 Jan 2008 17:39:42 +0000

whose the best RoR host out there in all of your opinions ?

By: Amanda

Amanda — Wed, 16 Jan 2008 17:13:48 +0000

I haven’t been a fan of Joyent since my company first started looking for a new hosting company a few months ago, with their horrible communication skills, pricing, amongst other things. Now they’re DEFINITELY off of the prospective hosts list.

By: dave

dave — Wed, 16 Jan 2008 16:51:34 +0000

Wow first Dreamhost, now Joyent with a major screw-up. BTW – they must feel pretty silly right now. Touting Sun for a couple years, pimping Sun hardware like it was the second coming and now they’ve switched to Dell and gotten smacked by ZFS.

By: anonymous

anonymous — Wed, 16 Jan 2008 16:23:17 +0000

Joyent’s responsible for a good part of twitter’s down times. Rumor has it twitter is moving off of Joyent as soon as they can. Sometimes marketing hype is just that.

By: doug

doug — Wed, 16 Jan 2008 15:24:54 +0000

I don’t understand the comments asking for comparisons between joyent and S3. S3 is a _storage_ platform, not a “cloud” computing offering. Perhaps you meant EC2?

Regardless, I’m not moving anything important to EC2 until SLAs are put in place and they offer some basic features such as … say … a static IP address that I don’t lose if I reboot an instance?

I’m all for bleeding edge when I’m looking at development, but if my business is relying on it, I’m still hosting with more tried and true dedicated server technology for my production environment.

By: John

John — Wed, 16 Jan 2008 15:13:15 +0000

don’t knock them, it’s a minor thing and they’re on it.

should be fixed in another 3-4 hours.

they’re good guys!

By: Jason

Jason — Wed, 16 Jan 2008 15:08:11 +0000

It was a ZFS bug that was fixed well before they had the issue. Not updating to the latest version of FREE software is ridiculous and irresponsible.

By: Amit

Amit — Wed, 16 Jan 2008 15:00:47 +0000

Alright then, I was just planning on signing up with Joyent. I might as well hold off, but there are some people that have some deep resentment for Joyent. Is this valid or is Joyent overall reliable (excluding the current issue)?

By: Technicle

Technicle — Wed, 16 Jan 2008 14:57:10 +0000

I fully support attributing the story to a ZFS bug – because that’s exactly what it was.

By: Ross

Ross — Wed, 16 Jan 2008 14:55:26 +0000

I find this quite funny after Jason from Joyent bashed me for one of my articles: http://www.host...xt-web-20-site/

I’ll let you read it yourself but in it I basically said Joyent appears to have problems with downtimes/customer service which Jason blatantly denied, maybe he’ll post an apology….

By: till

till — Wed, 16 Jan 2008 14:39:27 +0000

The article is slightly “off”. The headline should be, “Joyent Suffers Downtime Due To Running Outdated Software”.

If you read closely, that was the reason. Not because of something unforseen. Just because someone didn’t update, etc.. Always sucks having to deal with backups and shifting around lots of data. But it’s their business.

By: Jason

Jason — Wed, 16 Jan 2008 14:20:35 +0000

Frank54 I can almost guarantee you’ve never set foot in the Solaris platform, let alone done ANY work in ZFS. Get back to your Windows 2003 server.

By: Andrew

Andrew — Wed, 16 Jan 2008 13:42:21 +0000

is there a good 3rd party comparison for S3 vs Joyent

By: s3box

s3box — Wed, 16 Jan 2008 09:41:04 +0000

Nothing beats S3 — not even Google.

By: Technicle

Technicle — Wed, 16 Jan 2008 09:36:16 +0000

>plain-text transfer via smtp), why can people trust unencrypted storage?
plain-text transfer via smtp), why can’t people trust unencrypted storage?

and there’s no need for any answer — it’s not a question, rhetorically or not.