Receive webpage with ESP32

[MJU20]

Long time ago I used an ESP8266 to receive a webpage and filter out data from the received data.

With the ESP8266 this was done with an extra microcontoller.
First I connected to an SSID, then I did a ClientConnect, after that I used a ClientSendRequest with a "GET http://www.website.be/mobiel/page/R1R2\r\n\r\n"
That returned the content of the website that was located on that URL.
This I filtered down to data I've used in my project.

How is this done with the ESP32?
I have a NetworkComms connected to the WLAN-ESP32 component.
I can login to my SSID, but after that things become blurred for me

I've tried connect to an URL at port 80 (the website is HTTPS). I use the URL as a string in this component.
The IP address of the website is not consistent.
If I type the IP address for that website (IPlookup), in my browser, I get an error.

So, how can I receive data from an URL like this:

Code: Select all

https://www.timeanddate.com/counters/fullscreen.html?mode=a&iso=20260202T00&year=2026&month=2&day=2&hour=0&min=0&sec=0&p0=337

What steps must I use to get the ESP32 to receive the data that a browser would receive?

After that I want to filter out certain data, but first I need to receive the HTML..

Thanks for any help!

[BenR]

Hello,

I beleive you would use something like the following.

SocketConnect("www.timeanddate.com", 443)
SocketOpen()
SocketWrite("GET /counters/fullscreen.html?mode=....\r\n", Len)
SocketWrite("Host: timeanddate.com\r\n", Len)
SocketWrite("Accept: */*\r\n\r\n", Len)
SocketRead(Response, MaxRxLen)
SocketClose()

I beleive the ESP32 should be able to handle HTTPS and also domain IP lookup but I could be wrong. Might be worth trying to find a http equivalent to be able to use port 80. Https port is likely 443.

[chipfryer27]

Hi

I've been playing with databases etc recently and you can capture a website reply with the ESP32 quite easily. I have the Network Comms component connected to the WLAN ESP32.

I used the Network Comms component to send a GET which requested a value from a hosted database, the request being handled by some PHP. If I point a browser and make the request I just get my info returned (e.g. FC1=12345). However there is a lot more HTML going on. The site sent the few bytes of interest in the Body and the actual received string was around 426 bytes with only ten of interest to me. These could easily be parsed out.

After I sent my GET I had a short delay which may not be necessary depending on timeouts.
I then used the Network Comms:Receive to capture the incoming bytes as a string.
The captured string can then be parsed

Regards

[MJU20]

Chipfryer, my hero!

Could you post an example please of what you are working on?
Thanks!

[chipfryer27]

Hi

It will form an update to this post https://flowcode.co.uk/forums/viewtopic.php?f=3&t=2248 and I'll try and do it later today.

Regards

[chipfryer27]

Hi

Whilst my code for an ESP32 worked in simulation, when I tried to compile it failed with various errors that seemed to relate to various string functions. Noticing that idf-5.1 is now the recommended version, resolving some compilation issues, I entered that rabbit hole....

Installation failed and it appears related to trying to configure Windows Defender, something I don't use on my W8.1 machine, although it is present.

So, I'm sure you can guess the fun I'll now be having trying to get things working again.

Regards

[Visse]

Hello i am new o this forum and also i am a beginner with the ESP Coding. I work on a project that will be integrated on the ESP32. I also want to view the data that i show on my external display on the webpage browser. At this point i can only show one value. I don't know how to show the other values. Can anyone help me with this problem ? Or some model that i can follow ? Thx .
This is my project.

ESP32.fcfx

(31.38 KiB) Downloaded 57 times

[chipfryer27]

Hi

First off I'd like to say that for a beginner I think you have done very well so far.

To display the other values is quite easy.

First in the Web Server component you need to tell it how many substitutions you wish to display (increase from 1 to 4) and these are indexed starting at 0.

Next in your chart add in more Webserver1:SetOutValueFloat(x,y,z) components where x=index, y=value you want to display, z=decimal points)

Next modify page 1 HTML to suit (e.g. Temp BME = %1'C etc etc) where %1 = index 1 etc.

I made the modifications for you so you can compare, and when simulating I can point my browser at my address and see the values.

ESP32_Modified.fcfx

(32.07 KiB) Downloaded 64 times

Regards