Talk about a serious outage. Payment gateway service provider Authorize.net has been down and out for several hours, a number of tipsters inform us. That has big implications: since the service is used by tens of thousands of e-commerce vendors to accept credit card and electronic checks payments on their websites (example), it likely means millions are being lost during its downtime. PayPal and Google Checkout are still up and running.
It’s unclear when the downtime started exactly, but the consensus is somewhere between 5 and 7 hours at this point (11 AM Eastern), with e-commerce vendors desperately looking for ways to contact the company or get any first-hand information about what’s going on and when the problems will be resolved. Twitter, meanwhile, is buzzing with the news as the United States wakes up (hashtag #authorizenet).
According to some threads in hosting forums, which remain unverified for the moment, there was a fire at a Seattle datacenter during this U.S. holiday weekend which caused a massive technical failure.
We’re trying to get more information about the situation.
Update: nobody is picking up the phone at the U.S. offices of CyberSource, the holding company of Authorize.net. Someone I talked to at their UK offices couldn’t help me and told me I should keep trying the U.S. office.
Update 2: Nathan Cheeley writes:
A fire in Fisher Plaza, Seattle has cause a massive power outage causing leading IP-based payment gateway solution Authorize.Net to go down around approximately 11:15pm PST (last night).
A traffic reporter for KOMO News that operates out of Fisher Plaza tweeted that a fire set off the sprinkler system which fried the generators.
Update 3: Authorize.net has set up a brand new Twitter account to keep everyone updated, confirming the cause of the failure was a fire but also stating an ETA for resolution is not available at this time.
Update 4: a new tweet says backup was in place but that datacenter was impacted as well.
Update 5: tweet number 4 says “transaction processing is back up, with the exception of Global processing.” (12 AM Eastern)
Update 6: Or not. (12:30 AM Eastern)
Update 7: Authorize.net reports that full transaction processing has now been restored with Concord EFS.
(Thanks to everyone who sent this in)
We are telling our customers to use Google Checkout or call us in the meantime. What a crazy thing to happen on a holiday weekend.
This maybe a good place to ask you guys –
We are looking for a hosted payment solution that allows full customization. So, PCI compliance is taken care of by the payment solution firm, but the payment page can still be customized to match our site’s theme. Do you guys have any recommendations for this?
I heard that they were launching fireworks INSIDE of the data center!
PaymentsGateway.net offers exactly what you’re looking for. If you go to their developer forum at: http://forum.pa...thread.php?t=48 take a look at their “Secure Web Pay” solution. Additionally, you might also take a look at their CMI solution for tokenized payments which allows you to make calls to store and charge credit cards (and e-checks) on their platform without having to store the sensitive payment information on your server. This takes you out of scope for PCI but still allows you to charge, refund, void, etc. the cards through API calls.
Since you asked…Commerce Lab from IP Commerce offers a hosted payments page that can be integrated to your existing process. http://commerce...yment_Page.aspx
The Commerce Hosted Payments Page presents a seamless payment process based on your CSS allowing full customization with your branding (or theme) and is a PCI-certified payment system.
This might bee a serious catastrophe, I think paypal is the payment gateway of all the time, great service for a lower price and 100% uptime, and for the “free people” nice supoort also, c’mon Authorize.net its pretty dam bad really.
John
http://www.encu...ntry/JobUsa.htm
————————-
US Employer
True, a very BAD thing to happen during the weekend. But the guys have rather handled it well and we hope the site is fully up and running soon.
Yeah, definitely sucks for people who rely on them for 100% of their payment processing. Just goes to show that any site accepting credit card transactions really needs to have a backup solution like PayPal or Google Checkout when something like this happens.
No doubt – Lesson learned in a big way. Installing PayPal backup now for future situations.
Wow, the title suggests that the company just went under, as in failed, bankrupt, etc. But the article says that there is a supposed data center issue. Big difference, especially in todays economy, when many businesses are actually failing.
You’re right, I updated the title.
Huh? Title’s pretty clear. Service isn’t functioning, so it’s down. Service is down, so clients are left hanging…
There’s no sensationalism at all in the title.
Ah, I saw it post-update, hah. Ignore me.
/me steps slowly away from the keyboard…
Jeremy, maybe in the future assume the poster made a legitimate comment and save your asinine snarky post for yourself.
I thought the same when I read the title. o.0
We switched to simple validation in our MIVA shopping cart, which just collects thw credit card info. Hopefully, we can charge it later.
This is not PCIDSS compliant, storing credit card information on your internet connected web server can get you fined up to half a million dollars by Visa and MasterCard.
That isnt exactly true. You can store payment info assuming you meet other requirements of the PCIDSS about encryption and network protection. While every effort should be made to reduce or eliminate the storage of payment information and consumer information where not required there are cases where a company must store this data. I think this would be a case where a company should be collecting order for later processing. Any good order management service should have payment processing fault tolerance built into it.
Why store it at all? Why take the liability?
PCI Compliance does allow you to store CC info on your webserver, with the following caveats. The Primary Account Number (PAN) that appears on the front of the card can be stored. So can the cardholder name, service code and expiration date. These cannot be stored indefinitely. Usually only long enough to allow for returns or disputed charges (60-120 days depending on credit card used). Also, the info should be encrypted and your server requires a firewall and anti virus protection, as well as limited access to your server; that should be password protected. Paper records need to be shredded and/or burned after storage time limits exprire. CC numbers stored on paper with only last 4 digits appearing are ok.
The full magnetic stripe, chip, CAV2 CVC2 (number that appears on the back of the card) and PIN cannot be stored.
Hey Rober, good synopsis of the rules, thanks. I was recently trying to wade through a document about an inch think on PCi compliance put out by the bank and it was totally unclear
Read somewhere else – they expect everything back up by 10 am PST…
Lousy title. Should have used the word “outage”. But a blog’s gotta do what a blog’s gotta do.
Any merchant who relies entirely on real-time processing and/or doesn’t store declined orders in their database isn’t taking their business seriously anyway.
What’s worse is many of these smaller merchants and home based business’ are currently on vacation and therefore won’t find out about all of these lost sales until Monday! Thankfully as it is a holiday weekend sales are slow to begin with.
Had a hard drive crash this morning, turns out the untested backup program doesn’t actually work, so I lost the reference IDs I need to bill about $400/month in subscriptions.
On top of that Authnet goes down, so I have to update 4 ecommerce sites to only accept PayPal for the time being.
This is looking to be a really, really bad day
What happened to redundancy? You would think that a company as big as Authorize.net would have some sort of data backup / failover system so this could not happen..
That would cut into their margins. Might as well have a single point of failure in this economy.
I agree – the idea that one location having problems can bring it down is just mad, considering how large they are.
Customers could have the process go through, as the payment information is automatically stored and charge the information later….
Yes, because it’s totally legal and Visa/MC won’t mind a bit if authorize just holds onto that info for a little while.
Another company where it seems impossible to get a hold of someone on the phone…?
They’re bringing the world of internet quality to customer service.
I understand that there was a fire but why could they have not issued a press release to let us all know? Instead, thousands of us lost sales and customers and had to figure it out for ourselves. No working phone numbers, no working websites and apparently CyberSource corporate has the day off and didn’t think this was important enough to send people in to answer the phones. I called and got a message that the office was closed. This is despicable.
This not only affects e-commerce but also many Card Present/In-Store operations. Authorize.net has a card present API used by many retail systems.
Awesome timing with the holiday weekend.
So much for geographical diversity and hot standby for critical services providers.
I suggest that every mid market business take their incumbent distributed server based point of sale gateways and messaging queues, and throw it all on a PAAS platform owned by a company that is that is actually and truly negative cash flow and in debt to their managed host – oh yes, Rack Space, or Amazon, who you cant complain to.
Go do it. Oh, they admit that yes, they are down, and no, they have no idea…..and yes, you will get a credit, and no, you are not indemnified against business loss due to the outage.
Glad we use trustcommerce.com
Fire at the Fisher Plaza in Seattle. Not only has it taken down authorize.net, but also geocaching.com, plus probably others. A TV and radio station also have been affected and are using alternate transmission sites.
http://www.seat...plaza_fire.html
interesting that 3+ years after acquiring authorize.net cybersource did not integrate them into the main cybersource datacenters, which they have multiple and is the system that processes all their medium and large customers.
Hmm… Conspiracy Theory – CyberTerrorism is trying out various things: first rackspace, then app engine, and now Authorize.net
Hasn’t this week seen some action…
This is not the first problem at this building. Fisher Plaza previously experienced major outages due to electrical issues in 2008 and 2006.
Transactions are still not working as of 09:20 PST.
Looks like the Authorize.net gateway is back up.
Authorize.net has reported that transaction processing is back up.
In a short statement issued just a few minutes ago, a representative of the gateway said “ I understand transaction processing is back up with the exception of Global processing. We are working to bring that up ASAP.â€
Follow the latest development on Authorize.net Twitter page at http://twitter....om/authorizenet
http://twitter....om/authorizenet
No… the Authorize.net gateway is not back up, regardless of what they say
Nettica DNS service down too – they host out of Fisher Plaza and their homepage redirects to their news blog right now http://www.nettica.com
The whole point of a managed DNS service is to add/help with redundancy on top of content. Check out http://dynect.com and http://204.13.248.102 (watch video on how to Break Free).
our transaction began hitting again a few minutes ago.
Has anyone successfully cut over to an alternate provider? We don’t have any ecommerce but we do a lot of Virtual Terminal stuff and we’re stuck as well. PayPal offers a Virtual Terminal but it takes 48 hours to open an account.
I use Trust Commerce like the other posts….used Authorize at one time. The switch was easy. I’ve had nothing but good experiences with Trust.
Hope they will recover fast!
Wow
The economy is down already – do we need more problems
Definitely not a fun morning for many merchants and customers too.
This brings up two very important issues…
1 – The value of having alternative payment options for crashes and users who prefer to use them (paypal, google checkout, phone, etc…).
2 – Having good error messages. Many commerce sites just say “error” or have their site timeout when these issues come up. This doesn’t instill any trust with the customer who isn’t sure if they got charged, should reorder or what to do.
Give people options & information.
I still can’t log on and process through virtual terminal.
why don’t they have their own generators and back-up gas supply like our data center does? Ours can go for 2 days without electricity using their own. Or was the fire structurally damaging?
Google Checkout needs to lose the “registration” process if they want anyone to really use it.
From: http://www.meri...g/msg19026.html
Fisher Plaza, a self-styled carrier hotel in Seattle, and home to multiple
datacenter and colocation providers, has had a major issue in one of its
buildings late last night, early this morning.
The best information I am aware of is that there was a failure in the
main/generator transfer switch which resulted in a fire. The sprinkler
system activated. From speaking to the fire battalion chief, I am under the
impression that Seattle Fire did use water on the fire as well, but I am
unsure of this.
Given the failure location, generator power was not available, and cooling
failed. UPS power to systems continued, and I can personally vouch that
they held out for well over an hour. When we were able to access our
equipment, ambient air temps were well over 100 degrees in the room our
equipment is located in.
At least some, if not many circuits were affected. Several large
co-location providers and other datacenters are located in the facility,
these facilities have no power.
As this was the main/generator switch, and it is now highly damaged, the
circuits in the area are damaged, and the entire area is doused in water, a
rapid restoration of power does not seem likely. Fisher Plaza’s phone
numbers now result in fast-busy signals, so I have no recent update from
them directly.
Interestingly, this building is also the production studios for several
Seattle TV and radio stations.
There is no ETA for resolution.
I feel bad for authorize.net customers, and at the same time I am glad we switched to Braintree Payment Solutions last year instead of authorize.net.
We contacted authorize.net in Nov 2008 and they forwarded us to some merchant account company I had never heard of. A week later they couldn’t answer my questions and wanted me to contact authorize.net about those, I was turned off by that and by their discount rates.
I had heard of Braintree on hacker news so I contacted them. We went live with Braintree in January and they are awesome. My account rep answers questions related to payment settlements and she has even helped me troubleshoot some api issues I had.
If anyone is considering switching because of this outage, you can find tons of advice on hacker news by searching on searchyc.com.
My Authorize.net seal was causing my site to load slowly, so I solved the problem by disabling the seal.
I’ve switched my payment settings so my cart will accept credit card information. However, it will not automatically authorize or capture a sale. This seems to be working fine.
authorize.net back up for me now at 1:30pm EST
I was able to finally log into Authorize.net – 1:50pm – but when I tried to add a new order to the Virtual Terminal – the transaction would not go thru – for unknown errors. Will wait a little while.
Here’s my affiliate sign up link for authorize.net. Don’t all of you jump on it at once!
As a network admin I am totally blown away that a company we entrust with our lively-hoods does not have the necessary redundancy built in to their systems. I blame myself though. The thought that they didn’t have a disaster plan to meet every scenario imaginable never even crossed my mind. I’m sure they would have told me this could never have happened even if I had asked though.
I would bet that a cost cutting management team ignored the recommendations of the Admin team, who I am confident were pushing for more redundancy if they were competent. I know Admins can sometimes appear to be over zealous but this is the reason why. It is very stressful managing a system that is not prepared for catastrophic failure do to management oversight.
I hope other managers can learn from this what the true cost/benefit of redundancy is.
You’ve nailed it. Yesterday most of us would have assumed that Authorize.net was, ahem, “too big to fail”. But that very assumption reveals the same problem in our IT processes that seems to have just been revealed in theirs.
Authorize.net back up for me at 1:45pm EST. Don’t know for how long what with everybody trying to send all their “held” transactions at once, but we shall see. Will be looking for backup or alternative method of taking payments for future issues though.
I think it is working now our payments are going thru . so make test transaction , but authorize.net site is still down
Leo
What a crazy thing to happen at all… “banking stuff” should not rely on one physical location… I have more security guarding my personal photos…
1:30 p.m. CDT, Authorize and Global back up for us, both for order processing and remote/bulk capture.
Once again the importance of redundancy, failover, and GSLB comes into focus. How many more outages like this are going to happen before all companies outsource their DNS?
http://dynect.com
http://bit.ly/McUTc
Check out rocketgate as an alternative payment processing company. RocketGate uses multiple live active datacenters so there is no ‘failover’ needed to keep your business up and running
my current employer uses authorize. it’s API error documentation is horrific and you can’t manually hit customers in your vault. both a stark contrast to braintree payment solutions which i have used in the past – highly recommended. no i don’t work there.
Hopefully, this is a wake up call for all payment processors to re-evaluate their backup plans. Thankfully, our merchant CDGCommerce had a cost-free alternative Quantum Gateway for us to utilize for the time being.
In the last 7 hours, there have been approximately 340 tweets using the #authorizenet hashtag.
The situation, in general, indicates several things…
– Payment acceptance is the lifeblood of the small business.
Not a surprising piece of information…but something that is oft forgot when discussing payments acceptance and initiation modalities in general
– Redundancy…redundancy…redundancy
As someone who lives in the payments space, I am intrigued with the information that will flow out over the next few days regarding the situation. A fire in the datacenter is simply a nightmare…but, why did it take so long for customers to be notified? Why wasn’t there a fail-over to a backup site?
I suspect, believe, that true, full analysis of the situation won’t ever be fully released. But, as information trickles out, it will be intriguing.
– PCI-DSS education is still necessary
As emphasized by some of the comments toward the top of this list…Simply capturing the payment information for “future” processing (in cleartext) is wholly unacceptable. And yet, due to the situation, merchants are faced with the distasteful task of either putting their customer data at risk or losing sales.
An unfortunate situation…wholly unpleasant…and somewhat intriguing.
Thanks for the link! Glad I could help.
The first thing I did was look for @AuthorizeNet on Twitter. Their account was there but no followers or updates. I see now they must have just made it a little bit prior to my search.
It’s amazing how Twitter is helping to break big stories quicker than ever. It’s just a matter of finding credible sources to back up the tweets. Great job as always TC.
So long as you’re setup to save both abandoned carts & the customer info from non-finalized orders, you should be able to follow up by email or phone and get payment info after the fact.
But of course preemptively notifying the customer while in checkout and having them choose an alternate option such as “mail in payment” would prevent said customer from receiving an authorize error message in the first place.
Authorize.Net Web site at http://www.authorize.net is back up and seems to be responding quicker than before.
Authorize.net is supposed to have a redundant backup system. Why didn’t it work. Does this call for a class action suit?
That is a serious outage, but non-SaaS enterprise apps experience similar outages, and to one degree or another, these things are to be expected.
Sure everybody should be concerned with availability (and “scalability”) when they select a SaaS vendor. I reduce risk by only playing with the “big boys,” but they have gone down from time to time as well.
The question is not whether there will be some downtime, but whether a company, especially a small to mid-size company, is in a better position than a trusted SaaS vendor to manage the app in question?
Whether or not a businesses operates in a cloud or not is a irrelevant. Both models still operate from physical locations. The issue is redundancy plain and simple and down time is unacceptable for any mission critical service. Don’t take my word for it, sit in on a disaster preparedness class that all Network Administrators should have taken and you will find that there is no mention of acceptable down time.
However; if your argument is that “these things are to be expected” because of incompetence, I agree. They should not be the norm but rather the exception though and if we give them a pass on outages such as this then we get what we deserve.
We were one of the businesses hit. Luckily for us, our customers had cash.