Hosting provider SoftLayer was partially taken down this morning from a DDOS attack, and several well known websites, including TechMeme and TwitPic, went down with it.
The problem at SoftLayer was resolved, but some users of OpenDNS, a DNS service provider that is becoming more and more popular, still can’t reach those websites. The reason? OpenDNS caches IP addresses for domain names on a user’s computer, and they’ve cached a bunch of bad DNS entries now on these computers. This speeds up web surfing considerably, and has helped some users avoid major outages at the ISP level in the past. But in cases like today, with outages at the hosting level, the bad IP information ends up being cached for up to a day.
Users who know what’s going on can reboot their computers to clear the cache, but that’s clearly not a good overall solution. OpenDNS says they are turning on a feature called SmartCache that caches both the current and “last good” IP address, so situations like today won’t be an issue any longer.
DNS is the shame of the internet. starting with ICANN
i admit that’s a strong claim
oh, i agree. ICANN is more worried about increasing revenues that Internet stability.
eeek didnt knew that Softlayer got DDOS attack, damn i restarted my server in the morning after getting confusing why the websites were not loading and the issue continue for more than 30 minutes. I expected a email from Softlayer if there was some problem at their end.
Once that drama was completed i had to contact my server support company in order to know why all the websites were redirecting to open dns error pages and now this post is the answer to the same.
Thanks Mike!
Can’t they just issue a command to clear the cache? Sounds like a bad issue to me.
ok hang on, ‘CLEAR THE CACHE NOW!’ did that work?
computers are hard.
Windows:
c:\> iipconfig /flushdns
OSX:
dscacheutil -flushcache
Linux:
(If you need to ask, you should not be running Linux!)
should put that up in the post. lots of ppl have no idea you don’t have to reboot to flush dns.
Man, it is just one line: /etc/init.d/nscd restart
And that’s why people are afraid to switch to Linux
Its /etc/rc.d/init.d/nscd restart in Linux
In OSX Leopard its lookupd -flushcache
Not only can we clear the cache when a record is invalid, but so can our users. http://www.opendns.com/cache/ allows you to both check the results we are currently handing out and clear the cache if they appear incorrect. It’s a great tool for users and webmaster alike.
The downside of OpenDNS is that they show crappy unrelated ads when you are trying to use the address bar to search in FireFox.
Why would use the address bar? The SEARCH bar is right next door. Besides, those ads are minimal and configurable.
I have Firefox configured to search just from the address bar like Google Chrome does it.
If you search the intarwebs you will find numerous posts discussing how to change your settings with opendns to allow this function to work properly
What? All DNS implementations cache locally, regardless of what DNS resolver you use. Or are you talking about how OpenDNS caches IP addresses themselves?
This post makes no sense.
That’s what I thought. Is there something I don’t understand?
How is OpenDNS caching on my local box in any way that’s different from my ISPs DNS server?
Agreed. OpenDNS doesn’t install any software on your machine (it’s often installed at the router level), so perhaps the implication was that OpenDNS may impose a minimum TTL and doesn’t honor the original name server’s TTL?
Just so it’s clear, by ‘installed at the router level’. I mean OpenDNS name servers are configured and then that info is distributed via DHCP. Still no ’software’ of theirs installed.
Absolutely not. We always honor TTL and send a record with a TTL of 0 on domains that don’t resolve correctly (NXDOMAIN, SERVFAIL, no connection.) Some browsers cache this records for a small amount of time.
“The reason? OpenDNS caches IP addresses for domain names on a user’s computer.” …. as does every other DNS provider…it’s an OS/Application thing….not an OpenDNS thing. I’m surprised arrington.
On any Windows box from a command line:
ipconfig /flushdns
Takes all of five seconds. Similar commands for *Nx. Don’t know about Macs.
dscacheutil -flushcache
ty
Sometimes it takes an issue of this level to really poke a hole in an otherwise good plan. As a long time user of OpenDNS, the overall positives of the service definitely outweigh issues such as this.
The OpenDNS crew are usually pretty proactive; I’m sure they will iron things out.
“OpenDNS says they are turning on a feature that caches both the current and “last good” IP address” – they should have given a thought on this long ago!
This feature, called SmartCache, has been available since April 24th as an opt-in feature. It’s in the Advanced Settings page under the Dashboard’s Settings tab.
BTW, if any of you still can’t reach Techmeme, go to http://67.228.123.168/ instead! Or reboot your computer.
I had all sorts of bad luck on a host who configured my VPS to use OpenDNS – it seemed to go down constantly causing all kinds of havoc on my box.
I tried changing the DNS server many times and they’d reset it to the OpenDNS servers – it drove me absolutely nuts.
PS – ipconfig /flushdns didn’t seem to help me when I would reset the DNS servers.
Ha, funny, a guy who cannot deal with server problems, believing a DNS server could change the DNS server settings remotely, and the best, tries to “reset DNS servers” with “ipconfig/flushdns”. A good one!
No, ipconfig/flushdns clears the local resolver cache on a Windows machine. You had to go into the TCP/IP properties of your network connection to change the DNS server settings. There is no “reset” (to what?)!
Use google chrome & use search integrated in the address bar. Google is almost always the best solution.
…for full tracking of your surfing behavior and presenting the resulting optimized ads. Who knows what else they do with the data they collect from you….
See http://www.google-watch.org/ and other sources.
OpenDNS’s performance has taken quite a few hits in the last few days, I suspect this is due to their growth being well over 10 Billion queries.
Even using OpenDNS in the office today, accessing Google for hours became a task with connection failure messages every few hits, we ended up disabling it in the office for the rest of the day.
First, OpenDNS does not cache anything on a users computer. A computer’s operating system (and some browsers) cache recent DNS requests. Firefox, for example, will normally keep DNS responses for 60 seconds. The response we send when a domain fails to resolve (either due to a server failure or a non-existent domain) has a Time To Live of 0 seconds. These records are not normally cached by the operating system, however some browsers will cache these records for a small period of time.
Secondly, the issue with SoftLayer was only resolved 2 hours ago. SoftLayer put an appliance in place to mitigate a DoS attack that throttled OpenDNS due to our high number of users and, directly related, our high volume of legitimate DNS requests. I was the one to call SoftLayer, speak with someone in their IS department, and get the matter resolved. Only 2 hours ago, SoftLayer added OpenDNS to their whitelist, allowing our servers to query their nameservers normally.
In short, OpenDNS *did not* extend the downtime, SoftLayer did by accidentally blocking legitimate traffic from OpenDNS.
This is exactly why we implemented our SmartCache feature too. SmartCache will return the last good DNS result we got when an authoritative server fails to respond even if that record has expired. This feature is disabled by default, but can be turned on for any OpenDNS user by visiting the Advanced Settings page under the Dashboard Settings tab. You can read more about SmartCache at http://bit.ly/SmartCache and http://bit.ly/SmartCacheBlog .
you may want to compare notes with David and Allison.
Nothing David and Allison say is going to change the fact that this statement is objectively false:
Your article title states that using OpenDNS can lead to longer outages. This is also not true.
I agree, I use OpenDNS and I was alarmed by this article. After reading it I was a little skeptical as to how “OpenDNS caches IP addresses for domain names on a user’s computer” since I did not install any software from OpenDNS
It was not until I read the comments that I discovered the slant of the article was misleading. And not until I read the comment by Brian Hartvigsen that I was certain that the article was bad.
I think you owe it to OpenDNS to update this article with the real and complete facts, otherwise you are just bashing a legitimate service for no good reason.
Reminds me of the classic quote – “There are only two hard things in Computer Science: cache invalidation and naming things”
At least they got a good name…
One of my ISPs recently did temporarily use openDNS, and for a while I could not get access to let’s just say the more interesting sites.
I almost quit their service thinking they were trying to be my Mommy and Daddy.
Claiming that a problem resolved by restarting your computer is a problem with OpenDNS is exactly the kind of moronic drivel I have come to expect from you, sir. Nice work.
You should probably go ahead and retract this garbage.
second that
Third that.
Interesting that David U founder of OpenDNS hasn’t responded on this blog yet. He was so quick to respond when TC covered their last VC round but nothing on this yet….hmmm!
I tried OpenDNS last year for few weeks and in all honestly didn’t see any performance improvement. I think its just another marketing BS.
we’ve spoken a number of times this afternoon. he’s in his car and on his way back to the office, says he’ll respond then.
I’m still in meetings for the rest of the day — but if @arrington was totally wrong I would have told him as much on the phone.
I’m still catching up on email and I think that this is a case where we aren’t doing anything the ISP isn’t doing from a caching stand point, but we’re now a face (through our error page) to a problem that was previously anonymous (authoritative DNS outages as was the case here).
The irony is that our SmartCache service which any user can enable for free, handles this issue specifically and prevents it from being an issue. I think we’ll now turn it on by default.
I also find that OpenDNS is blocking sites that have been miscategorized by its amateur community and it is affecting other businesses. I no longer recommend OpenDNS if it begins to censor sites like this.
OpenDNS never ever blocks anything by default, except phishing and malware sites. This is the OpenDNS user requesting OpenDNS to block something against him. But I agree, the sites in the categories are often not tagged accurately. But nobody is forced to use this (optional) category filtering.
this article is just plan silly. i mean really… what does outage at any site have to do with specifically with opendns? regardless of who you have your resolvers pointed to the problem of caching will be the same. and it has been in my experience that benefits of opendns are numerous, when configured to meet one’s specific needs (and you know what you are doing). and opendns is much better than the isp provided dns service.
This post fails to cite my funny but incorrect first reporting of this
http://twitter....atus/2883497289
a few minutes later i tracked it down to opendns.
Mike,
This story seems a bit silly really. I saw the outage today and I didn’t have my downtime extended, while being a long time opendns user.
It should be noted that opendns users can check the cache on opendns servers and flush it: http://www.open.../support/cache/
it should also be noted that the cachecheck was showing no response for softlayer domains for seattle, Palo Alto and Chicago for a good portion of the day even after refresh the Cache was done.. seems like the rest of the Opendns servers were not being blocked by Softlayer at the time..
I have to respond here …
I do not understand all the hate towards open dns.
their service is configurable and easy to use .
first the service that is being bickered about is opt in .
second censoring ? thats is also opt in and configurable.
third performance issues ? im just one user but i have yet to have any problem with open dns resolving addresses.
fourth Advertising if you call it that … is unobtrusive and the results are ethical and link to reputable companies offering services and products based on the term.
and last
open dns , is in it self a opt in service you do not have to use it .
you do not have to pay for it and it is not forced upon you.
all the angst here would be better directed at the companies who you pay hard earned money to
for removing applications that their users clearly want.
for changing unlimited plans to capped.
to deleting media off your personal device.
and all the other real threats to your internet lifestyle.
the hate here is just plain silly and undeserved.
Well spotted! Seconding you. Thanks.
Mike,
Articles such as this damage the credibility of techcrunch.
If you are going to write about tech you should be much more rigorous in your understanding of how this stuff works.
Bad press for OpenDNS…why?
It’s always been a good service. This article feels a bit like an attack to me. Maybe that’s just TC’s way of hyping a story. Really unjustified in this case though, there are plenty of companies far more worthy of bad press, especially considering OpenDNS had a solution ready before the problem even occurred.
It also messes up Active Directory on MS Domained networks.
Does it? We use OpenDNS in our huge corporation, Windows AD, Unix, MF and more, some 250000 DNS lookups per day – no problems. It rather looks like you need some education on how AD has to be administered…
I’ve been using OpenDNS and have noticed that it always takes longer for them to refresh the DNS info. Websites that switch IPs are always last to resolve on OpenDNS. New domain names are always last to resolve on OpenDNS. It’s a useful service, but with ’speed’ as a selling point, it is rather slow in these instances.
I have SmartCache turned on, and I never noticed a problem.
*shrug*