Hi I was wandering if someone can help me, I am new to VBA and I wanted to create a scraping tool to extract some data. I have 2000 customer numbers in a table, I have to enter the customer number into our internal System and I have to obtain the customers details and other information from different screens. Once I have obtained this I have to copy it all to an access database. I was wandering if there is a way to create a BOT or Automating it using Access VBA?
Create A Scraping Tool using VBA (1 Viewer)
- Thread starter dassuz
- Start date
- Local time
- Today, 15:08
- Joined
- Jan 23, 2006
- Messages
- 15,379
Where exactly is the data you need to extract and what exactly is the format of that data?
If it's an internal table, why can't you get the data directly without having to "scrape" it?
Can you show us some sample data? and anything you have tried?
If it's an internal table, why can't you get the data directly without having to "scrape" it?
Can you show us some sample data? and anything you have tried?
The Data is located on our internal servers, it is located on our SAP Billing System, it is like any other system where you enter the customers reference number and you browse the different screens by clicking on tabs to view the data.
I have entered some screenshots in the attachment, I have had to remove some of the sensitive data. I am unable to get this data as I would have to request it through our centralised MI Team and could take weeks for them to extract the data I need
I have entered some screenshots in the attachment, I have had to remove some of the sensitive data. I am unable to get this data as I would have to request it through our centralised MI Team and could take weeks for them to extract the data I need
- Local time
- Today, 15:08
- Joined
- Jan 23, 2006
- Messages
- 15,379
Hmmm?
If you have a warranted need for the data that is resident/created/stewarded by your organization, then I recommend you speak to the "powers' within the organization.
Scraping screens and creating a "version" of such data for some "purpose in a local system or area of the organization" seems strange to me.
I have seen such activity where online system screens were "scraped" (actually captured and re-entered) into local spreadsheets and "manipulated locally" and reports created. This led to conflicting results in presentation to management. There is a reason for authoritative tables and processes. Online system was real time. The local system was point in time. Management soon learned the importance of "authoritative tables and real time processing".
I'm not saying your request is not appropriate. I am suggesting that requesting a copy or download of the required data (your management to IM) is a first step.
Good luck.
If you have a warranted need for the data that is resident/created/stewarded by your organization, then I recommend you speak to the "powers' within the organization.
Scraping screens and creating a "version" of such data for some "purpose in a local system or area of the organization" seems strange to me.
I have seen such activity where online system screens were "scraped" (actually captured and re-entered) into local spreadsheets and "manipulated locally" and reports created. This led to conflicting results in presentation to management. There is a reason for authoritative tables and processes. Online system was real time. The local system was point in time. Management soon learned the importance of "authoritative tables and real time processing".
I'm not saying your request is not appropriate. I am suggesting that requesting a copy or download of the required data (your management to IM) is a first step.
Good luck.
This responsive, user interactive, constant behind the scenes web calls and loading 500 MB of JavaScripts bs has made it so not everything is scrapeable any more. Get to a page of data you want to scrape and then view the HTML source. Can you find the data you want in there? If so, this is scrapeable, if not it's not.
Scraping essentially gets that HTML source file and parses out just what you want. If the data isn't in there, then your out of luck in the scraping department.
Scraping essentially gets that HTML source file and parses out just what you want. If the data isn't in there, then your out of luck in the scraping department.
- Local time
- Today, 14:08
- Joined
- Feb 28, 2001
- Messages
- 27,146
If there is any way you could generate a report or get that tool to export a spreadsheet, you would do much better. Data scraping is in essence a problem in parsing. Reports and spreadsheet are easier to scrape because of the predictability of what they produce. XML or HTML require a lot of syntax-capturing tools, and parsing is NOT an activity for the faint of heart, particularly for someone who is, as you claim, new to VBA.
Can it be done? Probably. Is it optimum to do it this way? Probably not.
If you need this because someone in your department wants it, ask that person to talk to the managers of your data source to see if you could get a report or spreadsheet exported for your project. The higher up the food chain that person is, the more clout s/he can wield to get you a decent intermediate data format.
Can it be done? Probably. Is it optimum to do it this way? Probably not.
If you need this because someone in your department wants it, ask that person to talk to the managers of your data source to see if you could get a report or spreadsheet exported for your project. The higher up the food chain that person is, the more clout s/he can wield to get you a decent intermediate data format.
