MartyConnelly
martyconnelly at shaw.ca
Sun Apr 9 14:01:01 CDT 2006
Well here are tools that you may have to use, if you get stuck. TCPTrace, YATT or PcapTrace are essentially tools that log everything going back and forth on the wire. I had to use on an amazon uk bookstore site when they weren't giving me enough info and they were making initial development mistakes. Useful, if they are throwing things like logging or session cookies at you, without telling you. Simon Fell is generally about 6 months ahead of microsoft, especially with SOAP development. http://www.pocketsoap.com/yatt/ Jim Lawrence wrote: >Hi Marty: > >I guess this is one of the practical issues/problems with screen scrapping. >This is why I believe it could become a full-time occupation just to keep >the connections. The client wants start off using an existing API and maybe >move to custom one if that does not fill the bill. The client is >viewing/scrapping a specific set of sites. The sites that will be viewed are >ones that would want the general public to view them as easily as possible. > >So we will see. If there are any other suggestions that you have I would be >more than glad to hear them. > >Thanks >Jim > >-----Original Message----- >From: accessd-bounces at databaseadvisors.com >[mailto:accessd-bounces at databaseadvisors.com] On Behalf Of MartyConnelly >Sent: Friday, April 07, 2006 9:52 PM >To: Access Developers discussion and problem solving >Subject: Re: [AccessD] Access Application - per unit cost > >Just one thing to keep in mind about screen scraping sites, companies >have a habit of doing a complete >web site redesign every 12 to 18 months, not to mention minor monthly >changes which may prove to be a problem. >Also some sites are putting in methods that may defeat older scraping >methods, like logging in with >a graphic designed login code parameter, page redirects or in ASP.Net >you may not be able to see the screen source. >Some of these are designed on puropse to defeat screen scraping. Some >companies like e-Bay provide a free API >to make it easy to collect data. Some companies provide an extra paid >for service to ftp or download the data as an Excel file. >Yahoo financial does this. Some companies may switch entirely to a free >or payment based WebService like USPS. >You could be programming against moving goalposts. > >Jim Lawrence wrote: > > > >>Hi All: >> >>Good news. After the client and I sat down for a conversation we hammered >>out a good deal. We have managed to ink-out a contract that both of us can >>live with. He admitted that he does not trust programmers after being >>'ripped' in a couple of instances. >> >>Dealing with an old coder like me seemed calm things down and we have a >> >> >deal > > >>that may see a solid ongoing full-time contract coming out of this. >> >>Thanks so much for all your suggestions. They definitely help put things in >>perspective. >> >>Jim >> >>PS So there is something that you young punk programmers do not have. >> >>-----Original Message----- >>From: accessd-bounces at databaseadvisors.com >>[mailto:accessd-bounces at databaseadvisors.com] On Behalf Of Jim Lawrence >>Sent: Friday, April 07, 2006 2:44 PM >>To: 'Access Developers discussion and problem solving' >>Subject: Re: [AccessD] Access Application - per unit cost >> >>Thank you all for your suggestions. >> >>I have planned to verbally submit an appropriate counter offer and let the >>project go if there is no agreement >> >>Jim >> >>-----Original Message----- >>From: accessd-bounces at databaseadvisors.com >>[mailto:accessd-bounces at databaseadvisors.com] On Behalf Of Jim Lawrence >>Sent: Friday, April 07, 2006 7:42 AM >>To: 'Access Developers discussion and problem solving' >>Subject: [AccessD] Access Application - per unit cost >> >>These are interesting times. >> >>Yesterday a programmer who is working with my company went to see a new >>client as I was out of the city for a couple of days on business. >> >>The client would like a custom made complex piece of code that would >> >> >'screen > > >>scrape' from other web sites so his site would be able to present a list of >>the best prices... a good idea. We had previously agreed on principle but >>had not negotiated a price. >> >>The client presented the young programmer with his position. >>First, we were to be on an hourly rate or when ever the client called. >>Second, the client would pay the programmer directly and not go through the >>company. >>Third, the maximum rate he would pay would be $12.00 and hour as he had two >>'east Indian' programmers, who had submitted bids of $5.00 per hour but >> >> >felt > > >>that he should try and support local talent. >> >>.. and a rider.. >> >>The client wants all code made to be his and if we ever used similar code >>again, in another paying contract we would pay him 50% of gross profits. >> >>I must say, I am almost speechless but please feel free to comment. >> >>Jim >> >> >> >> -- Marty Connelly Victoria, B.C. Canada