Solved Question: Is it possible to automate the actions to navigate an on-line database?

[MisterChips]

Just a query as to the possibility, (can it be done?) of automating the necessary keyboard/mouse actions to interact with an on-line database.

Problem: I am a member of a country-wide club that has just had a centralised database built.
Whilst the database is very good, it has been designed to help the central administration, not club secretaries.
It takes a dozen or so separate downloads to get the info we need to do our business.
Requests for them to build a single report to allow us access the full info that we need to do this has fallen on deaf ears.

So: Is it possible to automate the actions needed to call these files down?
Something similar to an Excel macro or an MS DOS batch file springs to mind.
It is many years since I have got involved with this sort of thing.

I don't hold out much hope . . . but a little hope is better than none!

Any advice or thoughts on this will greatly oblige.

 

[Gasman]

I believe it is called web scraping.
@arnelgp has done a fair bit of that in the past.

Search here for his username and that phrase.
No doubt he will respond now I have linked his username.

 

[CJ_London]

short answer is you can use the sendkeys function to navigate around a website, clicking buttons to create the file then download it - but can be tricky. Or as Gasman says use web scraping if you are not generating files.

But when you say 'online database' what does this mean? is it actually a web app? or do you have an actual link to a database? A database simply means where the tables/queries are stored - they don't provide a user interface such as a web app. And if you mean a web app, often these will have means to use REST to extract data (perhaps files as well), you would need to consult the developer section of the web app provider

 

[MisterChips]

Thanks Gasman, I wasn't aware that such programmes were so easily available.
I should have been to the internet before bothering the forum.

To be honest CJ the peeps who built the system are not interested in any chat about their system, and certainly not if it is likely to cause them and work!
I don't have access to the system, only my club secretary has that, but I have watched over his shoulder and by navigating around their system using various selections, it is possible to download .csv files containing the info we need.
It is this that I hope to replicate.
But. . . none of this has really answered your questions!

I will trawl YouTube to see if there is anything threre that approxmates our needs.

Thank you for your time in answering my qquestion.
I now have something too work on. Thnaks

 

[theDBguy]

Perhaps if you can tell us which system you are referring to, we can point you to a reference, if available, on how to use REST to communicate with it.

 

[CJ_London]

peeps who built the system are not interested in any chat about their system

So this is a bespoke system, not an off the shelf product? If the latter, there should be help online, no need to chat

 

[MisterChips]

Perhaps if you can tell us which system you are referring to, we can point you to a reference, if available, on how to use REST to communicate with it.

 

[MisterChips]

As much as I would like you to look at the site, I don't think they would be at all pleased with me for doing so.
I appreciate your offer to delve into this for me, but I must respectfully decline your offer.
I will, however, look into REST to see if I can make anything of it.
Thanks again for your time. . .it is appreciated.

 

[Pat Hartman]

Just be aware that screen scraping is fraught with danger. Any time the page owners change anything, your code could break. If all you need to do is to download one file, I'm not sure I would bother but if you want the practice, go for it. This is not too hard. Although when I built some code to do it, it was to fill in the forms so I was placing values here, there, and everywhere.

 

[MisterChips]

As I am trying to click on a button to cause a download of a file, I have been experimenting using Opera.com using ScrapeStorm.
So far without any luck.

Is there a simpler programme about that could do this for me, or is it easy enough to write my own code?

 

[Pat Hartman]

@MisterChips You marked the thread closed. Does that mean you found a solution? I'm sure that others would be interested in knowing what worked for you. You are very new here but many questions are answered by our searching our forum posts. If people don't include their solution, the thread will be of no use to people who find it in the future. Welcome aboard

 

[MisterChips]

Sorry about that. My original questions was answered with a YES, so I closed the thread. Perhaps I shouldn't have done so as others added posts.
There wasn't anything in the search that covered my issue.
I have tried, unsuccessfully, to use ScrapeStorm, and wondered if there was an alternative that might suit me better.
Should I open a new thread?
This forum has been a tremendous source of help, and I certainly wish to abide by the proper conventions and usages.

 

[Pat Hartman]

You could have continued on with the thread since you didn't actually get anything to work.
REST is only a solution if the website supports it and since yours is bespoke, it probably doesn't.
Tools like ScreenScrape are what you use when the website provides no way for you to get the data otherwise. But the tool might not allow you to do anything except read what is on the screen. I couldn't find the website so i can't tell if it also lets you poke a button.

We are in terra incognita here. This is an Access forum so although we all know a little about a lot of things, you probably won't find an expert here who can help you with a specific product unless you get really lucky. I would suggest you look for a forum that supports ScreenScrape. The company may have help available for you.

If you make progress with a particular approach, start a new thread and we'll pick it up from there. Post a link to this one though.

 

[isladogs]

Just to clarify, you want to read info on various web pages - not write to them?

I've used web scraping to read info from a number of sites in the past few years e.g.
a) to get info about the current Office 365 version from an MS webpage
b) so clients can check if there are newer versions of my apps available for download
c) to get a list of addresses for a specified postcode
I have example apps available which demonstrate all of the above

How easy it is to do depends on how well structured the web pages are.
Any browser will allow you to look at how the page is organised which will be your first step
However, as Pat has already stated, the process may well break if the pages are redesigned.

Using REST will only work if the system has been setup to allow download of files

 

[Pat Hartman]

I think he wants to push a button to generate a download. So he needs the website to recognize the poke. There may be other selection options. The OP didn't post a picture so we can't tell what else he might need to fill in or select but he needs to write to the page or at a minimum push the button.