Webserver does not always serve files correctly

[TheKaese]

I'm no networking expert by anymeans. But I'm running into an issue that looks like only part of an http response is being sent. For example:

Note in this screenshot, the tail end of my template index.html is shown. The rest of the content does not appear to ever show up.

This is not always occurring, but I would say 75% of the time my ESP boots up I am unable to access the interface due to this issue.
Full code can be found under /src/WebClient.cpp in https://github.com/TheKaese/IP-ESP32-CAM

[MoJo2600]

I think I can confirm this issue. My ESP8266 is serving a list of animated .gif files from LittleFS. On some of the files, the transfer just stops. Sometimes the Webserver stops responding completely.

[jendakol]

Guys @MoJo2600 @TheKaese, see #770 - it might help you.

[MoJo2600]

I will test it and report back. But it sounds like it is the same issue.

[TheKaese]

I took a look at the "changes", and I'm not really understanding what @jendakol is saying in the updated readme.
The example for "Specifying Template Processor callback" doesn't appear to actually do..anything? It just appears to look for a file?
Anyways, the first example seems like a good starting point... though I'd be curious for a rundown as to what's different about this vs. what I was doing in my example. I mean, it seems like my example is just shorthand for the same thing without the extra header stuff. Maybe there's more going on in the background. Very curious what you find @MoJo2600.

[MoJo2600]

Ok, I'm not a c++ expert, but I tried to implement the changes.

My code for serving static files before:

void handleFileRead(AsyncWebServerRequest *request, String path)
 { 
   // send the right file to the client (if it exists)
   Serial.println("[WEBSERVER] handleFileRead: " + path);
   if (path.endsWith("/"))
     path += "index.html";                    // If a folder is requested, send the index file
   String contentType = getContentType(path); // Get the MIME type
   request->send(LittleFS, path, contentType);
   Serial.println(String("[WEBSERVER] Sent file: ") + path);
 }

Now it looks like this:

bool handleStaticFile(AsyncWebServerRequest *request, String path) {
  
  if (path.endsWith("/")) path += F("index.html");
  
  Serial.println(String("[WEBSERVER] handleFileRead: ") + path);

  String contentType = getContentType(path);
  String pathWithGz = path + ".gz";
  
  if (fileSystem->exists(pathWithGz) || fileSystem->exists(path)) {
    bool gzipped = false;
    if (fileSystem->exists(pathWithGz)) {
        gzipped = true;
        path += ".gz";
    }
      
    Serial.println(String("[WEBSERVER] opening file: ") + path);
    File file = fileSystem->open(path, "r"); // e.g. SPIFFS.open(path, "r"); 

    AsyncWebServerResponse *response = request->beginResponse(
      contentType,
      file.size(),
      [file](uint8_t *buffer, size_t maxLen, size_t total) mutable -> size_t {
        int bytes = file.read(buffer, maxLen);
          
        // close file at the end
        if (bytes + total == file.size()) file.close();

        return max(0, bytes); // return 0 even when no bytes were loaded
      }
    );

    if (gzipped) {
      response->addHeader(F("Content-Encoding"), F("gzip"));
    }

    request->send(response);
    Serial.println(String("[WEBSERVER] sent file: ") + path);
    
    return true;
  }
  
  return false;
}

But sadly, the result is the same. I still have the issue, that files are not served:

[zekageri]

Maximum allowed TCP socket connection on an ESP32 is 10 at once. You must controll the requests on client side to not exceed this request limit at once.

[MoJo2600]

@zekageri Thank you for this input. Mine is a ESP8266. I implemented some throttling into my application, but still the issue persists.

I think that I will revert to the default ESP webserver. Not only because of this bug, but because of another issue - I'm interfacing a WS2812B LED strip. Since I moved to the async webserver the first pixel is flickering red, which is usually a sign that the CPU is under too much load to get the timing correct. But this is nothing that this library has to care for :)

[zekageri]

I belive, on the esp8266 the max allowed tcp connections is like 5 or 6

[jendakol]

Too bad it didn't help you guys 😞

@TheKaese What I'm trying to say is that there is a problem with serving too big files from the ESP and this code can help with it. However, it looks like you are suffering because you have too many files, not too big files.

[TheKaese]

Maximum allowed TCP socket connection on an ESP32 is 10 at once. You must controll the requests on client side to not exceed this request limit at once.

I appreciate the feedback, however this is not the problem. I have a single client connected.

Too bad it didn't help you guys 😞

@TheKaese What I'm trying to say is that there is a problem with serving too big files from the ESP and this code can help with it. However, it looks like you are suffering because you have too many files, not too big files.

Definitely appreciate the effort and work you put into your issue. However, I'm just serving two or three small files.. I'm not serving a ton. I guess I have to drop ASyncWebserver 😭

[jendakol]

I have a single client connected.

I think that doesn't say anything about how many connections does the client creates (assuming you mean client like a web browser).

I like this library but it seems quite dead. My PR is hanging there for more than a year, with no reaction at all. WS client doesn't work, issues exist, no reaction... So sad this is abandoned. Unfortunately, I don't understand it well enough to take it over...

[Pablo2048]

I appreciate the feedback, however this is not the problem. I have a single client connected.

It doesn't matter how many clients you have connected - it's all about browser behavior. And modern browsers do parallel download for the elements found in HTML. I see two options here:

1. do lazy loading of images
2. use one big sprite image

[zekageri]

Yes, as Pablo2048 said it's not about the clients. It's about how many requests are mode from one client at a time.

If you have 11 static image elements on one page if you are lucky the esp didn't crash and just not serving the 11th image. Meanwhile if you have other script file calls on the head or anything else, it get's overwhelming for the esp.

For example:

<!-- THE DOCUMENT REQUEST ITSELF IS ONE REQUEST -->

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
     
    <!-- HERE YOU INITIATE ONE MORE REQUEST -->
    <link rel="stylesheet" href="someCSS.css" />

    <!-- HERE YOU INITIATE ONE MORE REQUEST -->
    <script defer type="text/javascript" src="someScript.js"></script>
</head>
<body>
    
</body>
</html>

This little document will initiate 4 request.
Why 4?

• Document itself
• one css file
• one js file
• one favicon.ico --> basic icon for the web, i think every browser will request this.

And it's get's worse when inside your js file, you request things at document load event with a get request, and it's just one client. On an esp32 the ASYNCWEB will let you connect 8 websocket clients. If those clients will request the same page on the same time, it causes chaos.

What can be done about that?

• defer your script requests on the head. It will make the requests after the page has been loaded.
• make your scripts as small as you can. You can minify the JS scripts ( there are sites for that, even with APIs ) and gzip them and serve them like this. ( note that if you serve your gzip files with .gz extension the safari browser will not know what the F you just sent to it )
• try to make a small DOM at pageload.

[jendakol]

For me, it was better to have few huge files after all, than more small ones. That's a problem with images though, I didn't have any (almost)...

[zekageri]

There is a problem with huge files too. If you try to serve a file that is more than 130KB some magic happens and your file is stuck half way there. So far nobody knows why. It looks like the problem is with the AsyncWebServer libary. :/