Login | Register 
FEATURES
APPLICATIONS
DOWNLOADS
EDUCATION
BLOG
WIKI
FORUM
STORE

web dat for password protected page

General discussion about anything TouchDesigner

web dat for password protected page

Postby flaclad » Sat Jun 01, 2019 12:12 pm

Hi,
I'm trying to using web dat to get data from an html page that requires authentication.
In that same page I have the user and password input forms and a button to authenticate.

Is there a way to fill the input forms and simulate the mouseclick on the button before fetching the data?
or a way to send user and password info to simulate the auto-authenticate behaviour of the browser..

this is the html of the input forms and button

<input class="form-control" type="text" id="usernameBox" name="username" tabindex="1">

<input id="userNamePass" class="form-control" type="password" name="password" tabindex="2">

<input class="btn btn-sm btn-orange" type="submit" name="submit" id="signInButton" value="login" tabindex="3">
flaclad
 
Posts: 6
Joined: Wed Jan 16, 2019 12:35 pm

Re: web dat for password protected page

Postby snaut » Mon Jun 03, 2019 2:48 pm

Hey,

it might be possible to go directly to the authentication page.
First, the url for the web DAT should be the one that is set in the html form tag.
Next, use a Table DAT as the first input to the Web DAT and in 2 rows and columns specify the username and password with the first column being the input names (like "username" and "password") and the second column containing the values for these input boxes.

Best
Markus
User avatar
snaut
Staff
 
Posts: 901
Joined: Mon Nov 12, 2007 6:21 pm
Location: Toronto

Re: web dat for password protected page

Postby flaclad » Sat Jun 15, 2019 8:03 am

Hi Markus,
thanks for the answer. I managed to login but then I realized that the data that i want to scrape from the web needs flash player to be loaded. Using the webbrowser component from the Palette (which - I assume - supports flash), I finally get to visualize - and interact with - the data as a TOP rendered image. Is there a way to export to a DAT that html code that the WEBBROWSER from the Palette is reading?
I saw in the palette another component WEBCOMP that seems promising but I have no clue about getting it working in this kind of scenario...
anyone can point me to a possible solution?
flaclad
 
Posts: 6
Joined: Wed Jan 16, 2019 12:35 pm

Re: web dat for password protected page

Postby snaut » Sat Jun 15, 2019 11:19 am

Hey,

I actually didn't know that the WebTOP supported flash - that's surprising to me.
The WebDAT is the "source view" to the webTOP to some extend. It doesn't handle redirects etc but shows the same HTML that the webTOP renders.

Do you have an example of the data source? I'd be interested in taking a look at it.
Generally though, if it is contained in a flash file, as far as i'm aware you would not have access to it via the page's html source.

Can you see the data if you look at the page source in your OS's browser?

Best
Markus
User avatar
snaut
Staff
 
Posts: 901
Joined: Mon Nov 12, 2007 6:21 pm
Location: Toronto

Re: web dat for password protected page

Postby flaclad » Sun Jun 16, 2019 1:48 pm

Hi Markus,
you are most probably right about flash support.
Probably it’s better if a take a step back and I try to describe the project I’m working on.
I need to get data from different sites, the one I’m struggling with is CAM4, the popular erotic cam website (hope none getting :shock: :oops: ).
After logging in, when you connect to one of the thounsands of available cam rooms, you can visualize - above the chat - the number of users connected to that specific cam room, that is the data I’m trying to obtain which is nested in regular html:

<li id="usersTab" class="tabChat">
<a id="tab2" class="nopop"> Users
<span id="totalUsers">465</span>
</a>
</li>

I attach a toe file with:
- a web top, a web dat, a WEBBROWSER COMP.
- a table dat with username and password for a CAM4 account I created
- few screenshot of a standard cam

As the cams are not available 24hours, the one I have linked in the attached toe could be not available if someone is trying to test it, so you should connect to https://cam4.com with a standard browser, click on any of the listed cam, get the correspondent link and copy in the web top and dat.

If I feed the web dat with a table with username and password as you suggested, I think that instead of logging in and connecting to the selected chat room URL, I believe the web dat loads the home page of my account that is https://it.cam4.com/totalusers

I succeded to navigate to the cam room, log in and visualize the total users, using the webbrowser comp from the palette, because it allows to interact with the website as in a regular browser but I cannot access the html under the hood or extract text from it.
any advise?
Attachments
totaluser.toe
(185.81 KiB) Downloaded 142 times
flaclad
 
Posts: 6
Joined: Wed Jan 16, 2019 12:35 pm

Re: web dat for password protected page

Postby snaut » Thu Jun 20, 2019 1:14 am

Hm, I would expect that you can login via the webCOMP and using Basic Authentication.

Sorry - not able to try that out though..

Best
Markus
User avatar
snaut
Staff
 
Posts: 901
Joined: Mon Nov 12, 2007 6:21 pm
Location: Toronto

Re: web dat for password protected page

Postby flaclad » Thu Jun 20, 2019 7:08 am

Thanks Markus,
I tried the web com with all sort of authentication but I think I miss the way of properly using this comp, as I can't succed in gettin any html data even inserting any no-login web address.
anyone can point me in the right direction?
flaclad
 
Posts: 6
Joined: Wed Jan 16, 2019 12:35 pm

Re: web dat for password protected page

Postby snaut » Thu Jun 20, 2019 11:46 am

Just saw that the Session get's not activated by default. You have to toggle the Active Flag first to make this work.

Will fix this for the next release.

Best
Markus
User avatar
snaut
Staff
 
Posts: 901
Joined: Mon Nov 12, 2007 6:21 pm
Location: Toronto


Return to General TouchDesigner Discussion

Who is online

Users browsing this forum: Google [Bot] and 12 guests