[AccessD] FTP

Jim Dettman jimdettman at verizon.net
Thu Mar 8 08:38:27 CST 2007


John,

<<I think if I run into that I will need to ensure that the file is locked
(open it for writing?) or add a second file that is written by the FTP
client that says "I'm processing".>>

  I think I might do the latter.  I have been using the technique that if
the file exists, simply don't send another.  That works fine.  What doesn't
work is that I have found that FTP does not lock a file during transmission.

  So on jobs where I have a really tight schedule cycle (say every minute)
and the file is rather large, the transmitting end is still sending the
file, but the receiving end already starts to process the file and ends up
with a partial file.

  A couple of ideas I had to work around that is:

0. A processing flag that says "don't touch anything in this directory".

1. Sending along a second file for each file at the completion of
transmission (A "OK to process" flag for each file).  This avoids locking an
entire directory. Optionally, I could include a CRC checksum in the file.

2. Using a ZIP utility to zip everything (and thus provide a CRC check).

  Right now, I'm leaning towards #1.  The problem with this is you needs to
control both ends.  #2 is the only thing that would work if you don't have
control of the other end.  Outside of this, so far I haven't found anything
that will let me control file processing if I don't have control of the
other end.

Jim.

 

-----Original Message-----
From: accessd-bounces at databaseadvisors.com
[mailto:accessd-bounces at databaseadvisors.com] On Behalf Of JWColby
Sent: Wednesday, March 07, 2007 4:56 PM
To: 'Access Developers discussion and problem solving'
Subject: Re: [AccessD] FTP

>Does the application handle a 'PickupBox' as well as a 'DropBox'

Each drop box by definition has a pickup and drop box.  Which is which is
defined by the direction flag.  I currently use separate directories on each
end if bidirectional transfers are required.  They aren't always required.
For example, we send files to the bank but they never respond in any way

ATM it just sends and deletes behind itself, logging any errors.  This all
started with the objective of sending files for the insurance client.  They
have no method of handling such a thing.  In fact they often require that
the file name be very specific.  The concept generally is a process / delete
kind of thing where if the file is processed it is immediately deleted in
preparation for the next file.  I imagine that from their perspective, the
specific naming conventions allow them to have multiple files in the
directory for / from different clients, DIS' file has DIS_XXX.Dat where XXX
is an account number or something like that.

Obviously there will be issues of timing.  These files only occur daily but
what happens if the requirements push the timing up to where the transfer
can't occur fast enough to move the file before the next drop into the box.
I think if I run into that I will need to ensure that the file is locked
(open it for writing?) or add a second file that is written by the FTP
client that says "I'm processing".

John W. Colby
Colby Consulting
www.ColbyConsulting.com

-----Original Message-----
From: accessd-bounces at databaseadvisors.com
[mailto:accessd-bounces at databaseadvisors.com] On Behalf Of Jim Lawrence
Sent: Wednesday, March 07, 2007 1:25 PM
To: 'Access Developers discussion and problem solving'
Subject: Re: [AccessD] FTP

Hi John:

This sounds really good. Does the application handle a 'PickupBox' as well
as a 'DropBox' and how does it handle files with the same name just
different dates and time?

Jim

-----Original Message-----
From: accessd-bounces at databaseadvisors.com
[mailto:accessd-bounces at databaseadvisors.com] On Behalf Of JWColby
Sent: Wednesday, March 07, 2007 6:59 AM
To: 'Access Developers discussion and problem solving'
Subject: Re: [AccessD] FTP

I have my client purchase the 3DFTP product.  I don't have any commercial
products, all of my stuff is consulting jobs.  

I am developing an access C2DbFTP.mda (library / framework) that does drop
box monitoring to upload / download files.  This application just watches
specific directories and if a file appears, moves it to a different
directory.  Monitor FTP dir / move to local dir, monitor local dir / move to
ftp dir.  This application does not DO ANYTHING with the files moved, it
just moves them, and logs the move.

The idea is to allow any application that needs such functionality to just
drop a file in a directory and expect that it will be moved to the right
place.  For example DISCO - my insurance call center client's application -
needs to FTP files from insurance companies to the local drive.  Once a file
appears from the insurer, the file is opened by DISCO and loaded into the
database.  Likewise the application needs to ftp files back to the insurer
and also needs to ftp files to banks.  Thus DISCO prepares "reports" (as I
discussed in a previous thread) and drops those "reports" into a specific
directory on the local hard disk.  The C2DbFTP client watches these hard
disk directories for files, and simply transfers them up to whatever ftp
address is associated with that directory.

My C2DbFTP application has a table that says:

For tblDropBox:
Drop Box PKID		'PKID of a record
LocalDir			'Local directory on the hard disk
FTP site			'FTP site
FTP PW			'FTP Password
FTP User			'FTP Username
FTP Dir			'Directory on the FTP site
Direction of transfer	'Direction of the FTP transfer
Log Table			'Table to log results to

C2DbFTP loads this table to determine all of the directories / FTP sites to
monitor and which direction the transfer has to go.  C2DbFTP simply polls
each "source" directory for files, and if any files are found, they are
moved to the destination directory and deleted in the source directory.  The
Direction of Transfer tells C2DbFTP whether the transfer is from an FTP site
down to a local drive or vv.

When any file is transferred, it is logged in the Log Table.  The log table
is nothing more than the PKID from tblDropBox, the file name, the date and
time the move occurred, and an error code in case the FTP failed for some
reason.

So ANY application can write a record into the tblDropBox telling C2DbFTP "I
need a transfer done", drop a file into the directory that is to be
monitored (if a send), and watch the log table for the file being sent.  It
can get files this way, and it can send files this way.  If it receiving
files, it writes a tblDropBox record to set up the transfers and watches
local directory for files.

What is done with the files are not the concern of C2DbFTP, just the fact
that FTP / local directories need to be monitored for files to move.  The
application moving files does not have to "know how to do FTP", all it needs
to know is how to write a record to the table telling C2DbFTP to move a
file, and provide a log table to watch for the move results.  The
application uses / creates the files, C2DbFTP just moves them.

I am using this with another client to NCOA process addresses.  The client
runs C2DbFTP.  He sets up an upload directory to be watched by C2DbFTP and
then drops files of addresses to be processed into that upload directory.
C2DbFTP watches the directory and uploads them to my FTP site.  Another
instance of C2DbFTP running on my server here at my office watches the FTP
upload directory (for that client) up on my FTP server and if a file exists,
downloads it to my local server.  My local server runs the file through the
NCOA address processing system which drops the result file into a specific
directory on my local server hard disk.  C2DBFTP is watching that dir and
uploads the files back to the FTP site for that client.  The C2DbFTP
instance running at the client watches for files in that FTP directory on my
site and downloads to the client's hard disk.  The client's database
application watches for files in that local hard disk dir and if one
appears, loads the file into the database for processing / merging back into
his address table.

As you can see, neither my NCOA processing application nor the clients
database application needs to understand FTP, all it needs to understand is
where to place the files to be NCOA processed and what to do when NCOA
processed files appear in a specific directory.  Likewise, my NCOA
processing application doesn't need to understand FTP, all it needs to do is
know where to place files and what to do when files appear in a given
directory.  BTW, my NCOA processing application also logs the files being
processed and generates billing information which is sent to the client,
billing them for the NCOA processing.  

So C2DbFTP can be used by completely different applications to process FTP
transfers.

I have a wrapper class for the 3DFTP API.  And of course each of these
applications could be custom programmed to do FTP directly, inside of each
application using my wrapper class.  And if that becomes necessary I can
certainly do that, but a "drop box" metaphor with a log table works quite
well much of the time, and can be implemented much more quickly.  Drop
C2DbFTP in place and run it.  Drop records in the table drop box, and start
dropping files / watching for files.  

John W. Colby
Colby Consulting
www.ColbyConsulting.com

-----Original Message-----
From: accessd-bounces at databaseadvisors.com
[mailto:accessd-bounces at databaseadvisors.com] On Behalf Of Bobby Heid
Sent: Wednesday, March 07, 2007 8:53 AM
To: 'Access Developers discussion and problem solving'
Subject: Re: [AccessD] FTP

John,

How do you handle licensing with that product?

Bobby 

-----Original Message-----
From: accessd-bounces at databaseadvisors.com
[mailto:accessd-bounces at databaseadvisors.com] On Behalf Of JWColby
Sent: Wednesday, March 07, 2007 5:22 AM
To: 'Access Developers discussion and problem solving'
Subject: [AccessD] FTP

I noticed at least one person discussing monitoring directories and FTPing
up/down.  For those that do that, what method do you use for implementing
the FTP stuff?  I have been developing similar capabilities using 3DFTP
(which I love and recommend highly) and an API to program to it.  
 
John W. Colby
Colby Consulting
www.ColbyConsulting.com


--
AccessD mailing list
AccessD at databaseadvisors.com
http://databaseadvisors.com/mailman/listinfo/accessd
Website: http://www.databaseadvisors.com

--
AccessD mailing list
AccessD at databaseadvisors.com
http://databaseadvisors.com/mailman/listinfo/accessd
Website: http://www.databaseadvisors.com

--
AccessD mailing list
AccessD at databaseadvisors.com
http://databaseadvisors.com/mailman/listinfo/accessd
Website: http://www.databaseadvisors.com

-- 
AccessD mailing list
AccessD at databaseadvisors.com
http://databaseadvisors.com/mailman/listinfo/accessd
Website: http://www.databaseadvisors.com




More information about the AccessD mailing list