Surviving Sandy
On Monday, October 29th, 2012, Hurricane Sandy battered the U.S. Mid-Atlantic states, devastating entire neighborhoods along the coastline, including downtown New York City.Businesses in lower Manhattan—where the East River and the Hudson River meet—were shuttered for days. For these companies, the challenges to remain in business were daunting. Businesses were out of commission for weeks, as water cascading into the sub floors of their buildings wiped out all the vital support systems located below ground level.
Publishing giant American Media Inc., with offices one block from the East River overlooking Battery Park and the New York Harbor, was one of the severely impacted firms.
Sandy’s surge unleashed twelve feet of water into AMI’s office
building, causing massive damage. While full generator backups were
located on the roof, the water surge knocked out all the cables for Con
Edison, New York’s power provider, and Verizon, the building’s telecom
provider. The cables were located three floors below the building lobby.
The lobby itself was under 3 feet of water.Additional damage was caused by eruption of diesel storage tanks, used by Staten Island Ferries, which contaminated nearby buildings and created hazardous conditions.
With seven weekly and ten monthly titles as well as special interest publications, closing cycles at AMI run every day. Over the last few years, its production workflow had been updated, first with SendMyAd for advertising production, and then with a full-blown enterprise DALIM ES project management system with a DALIM DIALOGUE Engine for softproofing and DALIM TWIST for automating the workflow.
In addition to its offices in downtown NYC, AMI also has a main editorial/production hub in Boca Raton, FL with additional offices in Nashville, TN and Los Angeles, CA. Ironically, prior to Hurricane Sandy, working with Blanchard Systems AMI had implemented a backup schedule to synchronize assets between its New York and Florida locations in preparation for Florida’s hurricane’s season. All the data and assets created in Boca Raton are synchronized daily with the NYC office and vice-versa. Synchronizing those assets allowed AMI to secure on-time magazine closings, which would take just days after Sandy hit.
The issue for AMI was how to make sure the company could function moving forward. It was obvious that things were not getting back to “normal” any time in the near future. The logistics were as complicated as mounting a military campaign, requiring leadership, ingenuity and flexibility that was provided by CEO David Pecker. Staffing, accessing hardware, and setting up shop elsewhere all had to be dealt with simultaneously.
Planes, trains and automobiles
“As Sandy was hitting Connecticut, Daniel Rotstein, SVP Administration hired a Gulfstream Jet and made arrangements for critical staff to be picked up by Limo and driven to the White Plains airport” recalls Robert Fenikowski, AMI’s Director of Publishing Technologies. “We learned later that theWhite Plains airport was shut down minutes after our plane’s takeoff.” The private jet, a necessity because all commercial flights were grounded, took more than 40 employees to Florida where, the next day, they would complete the regular, final close of weekly OK! Magazine and take up residence for months.
Two days after the storm—Wednesday, October 31st—staffers were still not allowed into the NYC building. The entire area resembled a war zone. “In what seemed like a split second, 450 employees lost access to their desks, computers and personal be- longings,” notes Fenikowski.
The downtown NYC site had functioned as a data center for the entire company, and although assets were uploaded to the Florida location, there was a host of essential publishing technology housed in the offices; file and editorial servers, proofing systems, production and editorial desktop computers, as well as financial and billing servers. After some maneuvering, employees received clearance from Mayor Bloomberg that allowed them to get into the building and retrieve what they could. Whoever had transportation and gas made the trip. They had exactly one hour.
Once the crucial hardware was removed from the building, the next
challenge was how to get it to Florida. All the airports were closed—the
best solution per Rob O’Neill SVP Operations was to put the hardware in
a vehicle and drive it all down to Florida. Two days after Sandy, with
shortages of every possible resource in the NY Metro area and limited
travel options, the company was fortunate to secure transportation. A
rented truck, van and car made the trip, scavenging whatever gas they
could find to leave New York. Rich Waltman (VP of Production) and
Fenikowski drove 1,481 miles, down to Boca Raton, in the ridiculously
short time of 16 hours—it usually takes at least 22.Within 45 minutes of arriving at Boca Raton, the DALIM SOFTWARE system was up and running, thanks to expertise of the Florida office-based IT staff—a positive sign, as all of AMI’s monthly titles are produced using the technology. But AMI also had as- set management, editorial, and other systems. “We needed all of these elements available to us immediately,” says Fenikowski.
At the same time, nearly 500 people were displaced. “They did not have a place to work and could not work at home,” says Fenikowski. “Some had their own personal tragedies. And, it was difficult to commute, with travel restrictions caused by an almost complete shut- down of subway, train and bus systems—and car use limitations.
The company relocated anyone who could make the trip to hotels in Florida, to work from the Boca Raton office. “We tried to recover workstations from our downtown headquarters and deliver them to new, temporary offices. But, eventually, we ended up purchasing new computers and setting them up from scratch so people could work,” says Fenikowski. “ We took the most important pieces from NYC and relied on our daily backups between Boca and NY. Unfortunately some backups were corrupted be- cause the power went down. We wondered if the sustainability of the business could be in question.”
“There was never a question about how to technically produce our titles. The biggest factor was how to move our people and supply them with the tools and access to the Internet. In Boca, my system was set up quickly and I was finally able to check on my family. We were lucky. We were not affected by any damage other than loss of power, which was restored two weeks later. But before that, I did not sleep for four days and went through a lot of stress,” recalls Fenikowski. “I was relieved that the DALIM SOFTWARE system was not disrupted. It proved to be a smart investment.”
Eventually, over time, when systems were secure, everyone returned to New York. “Some titles in New York were hosted by our print providers and we were guests for a couple months, in the company of other distressed publishers. We worked very hard to bring people back to our offices. Our company sublet some space from another publisher and moved almost all our staff back from Florida to temp office space,” explains Fenikowski. “We immediately upgraded to ES 3 and put it in a virtualized environment. We had to duplicate all the systems. It was not really the way to bring back systems from Florida and we still did not have access to our main NY office datacenter.”
“One of the lessons learned was how you can take advantage of technology to survive a crisis like Sandy,” says Fenikowski. “ At the same time, there is the whole human element involved, and many other moving parts. You can be prepared as much as you want, but there are so many other factors in play that can completely destroy your recovery.”
“AMI had past experience relocating their production from Florida offices, caused by past hurricane seasons as well as an Anthrax letter attack follow- ing September 11 events. None of these events ever prevented us from missing closing dates for our magazines,” recalls Fenikowski. “Cloud-based solutions and most disaster recovery plans focus on data, systems uptime and process continuity. But it’s hard to build a recovery scenario when, at the same time, there is a complete shutdown of an entire city or region, with all the vital infrastructure being compromised.”
“The technology worked, but it was frightening about how it could bring us to think about the existence of all parts of our business. If there is an- other storm tomorrow, I’d bet that the majority of the companies in New York affected by storm will have the same problem they had with Sandy,” predicts Fenikowski. “Lots of buildings are still closed and many companies have returned technically to where they were before the storm, but with first anniversary approaching in next few months, many of them still have not completed a full recovery. They can talk about technology, offer a roadmap and graphics, and create PowerPoint presentations. But they ignore the biggest factors: it’s the human factor and logistics. We still struggle because the building is not ready, but we know how to do things better. We know how to build the workflows, pushed majority of our systems to the Cloud, and revisited working relationships with vendors. We know how to move things in hours, if necessary.”
http://www.blanchardsystems.com/american-media-inc-relies-on-blanchard-systems/
