[AccessD] OT: But only Partly

Jim Lawrence accessd at shaw.ca
Wed Mar 21 21:01:15 CDT 2007


According the Instructor and 15 years DBA who is teaching the course I am
currently taking...One backup system is not enough. Many big installations
have as many as 4 backup and recovery processes implemented.

Jim  

-----Original Message-----
From: accessd-bounces at databaseadvisors.com
[mailto:accessd-bounces at databaseadvisors.com] On Behalf Of Gary Kjos
Sent: Wednesday, March 21, 2007 7:37 AM
To: Access Developers discussion and problem solving
Subject: Re: [AccessD] OT: But only Partly

Ouch. We used to do Disaster Recovery Firedrills back in my early
mainframe days when I was a computer operator. We had an arrangement
with another local company that had a similar hardware configuration
to ours that we were backup sites for each other. In those days, disks
on mainframes were removable from the drives themselves which were
about the size of a washing machine. The disk packs were about the
diameter of a LP record and the ones we used were about 8 inches tall.
We would take our disks or maybe it was just backup tapes over to this
other company and they would let us use their system over night and we
would attempt to run our orders and print the picking documents. Since
the hardware configuration was slightly different we had different
execution job control that referenced the hardware they had there. I
was mostly just along to carry stuff in the early days but later on I
was called on to run the stuff too. When fixed hard drives and online
terminals came along in about 1980 that ceased to be an option anymore
as we would have had to actually overwrite their files on the disk or
they would have needed enough empty space for us to load our stuff on
and as disk was failrly expensive in those days that wasn't a viable
option. So instead we concentrated on getting better covereage from
our hardware maintenance group. And we used our backup tapes pretty
often when stuff got corrupted and had daily, weekly and monthly full
backups for an entire year of generations, so we were really quite
secure and fully tested backup wise. Noplace I have worked since has
had anywhere near that level of backup. But hardware failed a lot more
then than it does now too, so we get lulled into a sense of security
that drives don't fail. But in this case it wasn't even a drive
failure that caused it, it was a human mistake.

We had an occurance of the "can't read the backups" here a while back.
It was a very bad thing. There had been a change to the backup
software itself and maybe the hardware too. I don't remember exactly
what the end result was as far as data loss - don't think we lost
anything - but we were down for an entire day - no sales entered.
Order takers had to write orders down on paper to be entered later. I
think our website still took orders as it's seperate but there were no
confirmations etc. It wasn't a total loss as some of that business
came to us in the following days, but some of those orders went to
other sellers instead of us and perhaps some of those customers went
away disgruntled too.

GK

On 3/21/07, Rocky Smolin at Beach Access Software
<rockysmolin at bchacc.com> wrote:
> Are you backed up?  Are you SURE?
>
>
>
>  http://www.signonsandiego.com/news/tech/20070320-0509-lostdata.html
>
> JUNEAU, Alaska - Perhaps you know that sinking feeling when a single
> keystroke accidentally destroys hours of work. Now imagine wiping out a
disk
> drive containing an account worth $38 billion.
>
> That's what happened to a computer technician reformatting a disk drive at
> the Alaska Department of Revenue. While doing routine maintenance work,
the
> technician accidentally deleted applicant information for an oil-funded
> account - one of Alaska residents' biggest perks - and mistakenly
> reformatted the backup drive, as well.
>
> There was still hope, until the department discovered its third line of
> defense, backup tapes, were unreadable.
>
> "Nobody panicked, but we instantly went into planning for the worst-case
> scenario," said Permanent Fund Dividend Division Director Amy Skow. The
> computer foul-up last July would end up costing the department more than
> $200,000.
>
> Over the next few days, as the department, the division and consultants
from
> Microsoft Corp. and Dell Inc. labored to retrieve the data, it became
> obvious the worst-case scenario was at hand.
>
> Nine months worth of information concerning the yearly payout from the
> Alaska Permanent Fund was gone: some 800,000 electronic images that had
been
> painstakingly scanned into the system months earlier, the 2006 paper
> applications that people had either mailed in or filed over the counter,
and
> supporting documentation such as birth certificates and proof of
residence.
>
> And the only backup was the paperwork itself - stored in more than 300
> cardboard boxes.
>
> "We had to bring that paper back to the scanning room, and send it through
> again, and quality control it, and then you have to have a way to link
that
> paper to that person's file," Skow said.
>
> Half a dozen seasonal workers came back to assist the regular division
> staff, and about 70 people working overtime and weekends re-entered all
the
> lost data by the end of August.
>
> "They were just ready, willing and able to chip in and, in fact, we needed
> all of them to chip in to get all the paperwork rescanned in a timely
manner
> so that we could meet our obligations to the public," Skow said.
>
> Last October and November, the department met its obligation to the
public.
> A majority of the estimated 600,000 payments for last year's $1,106.96
> individual dividends went out on schedule, including those for 28,000
> applicants who were still under review when the computer disaster struck.
>
> Former Revenue Commissioner Bill Corbus said no one was ever blamed for
the
> incident.
>
> "Everybody felt very bad about it and we all learned a lesson. There was
no
> witch hunt," Corbus said.
>
> According to department staff, they now have a proven and regularly tested
> backup and restore procedure.
>
> The department is asking lawmakers to approve a supplemental budget
request
> for $220,700 to cover the excess costs incurred during the six-week
recovery
> effort, including about $128,400 in overtime and $71,800 for computer
> consultants.
>
> The money would come from the permanent fund earnings, the money earmarked
> for the dividends. That means recipients could find their next check
docked
> by about 37 cents.
>
>
> Rocky
>
> --
> AccessD mailing list
> AccessD at databaseadvisors.com
> http://databaseadvisors.com/mailman/listinfo/accessd
> Website: http://www.databaseadvisors.com
>


-- 
Gary Kjos
garykjos at gmail.com
-- 
AccessD mailing list
AccessD at databaseadvisors.com
http://databaseadvisors.com/mailman/listinfo/accessd
Website: http://www.databaseadvisors.com




More information about the AccessD mailing list