Extend documentation of the HTML forms example. fhessel#81

Author: fhessel

README.md

@@ -73,18 +73,19 @@ git clone https://github.com/fhessel/esp32_https_server.git
 
 > **Note:** To run the examples (except for the _Self-Signed-Certificates_ example), you need to execute the script extras/create_cert.sh first (see [Issue #26](https://github.com/fhessel/esp32_https_server/issues/26) for Windows). This script will create a simple CA to sign certificates that are used with the examples. Some notes on the usage can be found in the extras/README.md file.
 
-You will find several examples showing how you can use the library:
+You will find several examples showing how you can use the library (roughly ordered by complexity):
 
 - [Static-Page](examples/Static-Page/Static-Page.ino): Short example showing how to serve some static resources with the server. You should start with this sketch and get familiar with it before having a look at the more complex examples.
 - [Parameters](examples/Parameters/Parameters.ino): Shows how you can access request parameters (the part after the question mark in the URL) or parameters in dynamic URLs (like /led/1, /led/2, ...)
+- [Parameter-Validation](examples/Parameter-Validation/Parameter-Validation.ino): Shows how you can integrate validator functions to do formal checks on parameters in your URL.
 - [Put-Post-Echo](examples/Put-Post-Echo/Put-Post-Echo.ino): Implements a simple echo service for PUT and POST requests that returns the request body as response body. Also shows how to differentiate between multiple HTTP methods for the same URL.
 - [HTTPS-and-HTTP](examples/HTTPS-and-HTTP/HTTPS-and-HTTP.ino): Shows how to serve resources via HTTP and HTTPS in parallel and how to check if the user is using a secure connection during request handling
+- [HTML-Forms](examples/HTML-Forms/HTML-Forms.ino): Shows how to use body parsers to handle requests created from HTML forms (access text field contents, handle file upload, etc.).
+- [Async-Server](examples/Async-Server/Async-Server.ino): Like the Static-Page example, but the server runs in a separate task on the ESP32, so you do not need to call the loop() function in your main sketch.
+- [Self-Signed-Certificate](examples/Self-Signed-Certificate/Self-Signed-Certificate.ino): Shows how to generate a self-signed certificate on the fly on the ESP when the sketch starts. You do not need to run `create_cert.sh` to use this example.
 - [Middleware](examples/Middleware/Middleware.ino): Shows how to use the middleware API for logging. Middleware functions are defined very similar to webservers like Express.
 - [Authentication](examples/Authentication/Authentication.ino): Implements a chain of two middleware functions to handle authentication and authorization using HTTP Basic Auth.
-- [Async-Server](examples/Async-Server/Async-Server.ino): Like the Static-Page example, but the server runs in a separate task on the ESP32, so you do not need to call the loop() function in your main sketch.
 - [Websocket-Chat](examples/Websocket-Chat/Websocket-Chat.ino): Provides a browser-based chat built on top of websockets. **Note:** Websockets are still under development!
-- [Parameter-Validation](examples/Parameter-Validation/Parameter-Validation.ino): Shows how you can integrate validator functions to do formal checks on parameters in your URL.
-- [Self-Signed-Certificate](examples/Self-Signed-Certificate/Self-Signed-Certificate.ino): Shows how to generate a self-signed certificate on the fly on the ESP when the sketch starts. You do not need to run `create_cert.sh` to use this example.
 - [REST-API](examples/REST-API/REST-API.ino): Uses [ArduinoJSON](https://arduinojson.org/) and [SPIFFS file upload](https://github.com/me-no-dev/arduino-esp32fs-plugin) to serve a small web interface that provides a REST API.
 
 If you encounter error messages that cert.h or private\_key.h are missing when running an example, make sure to run create\_cert.sh first (see Setup Instructions).

examples/HTML-Forms/HTML-Forms.ino

@@ -43,9 +43,10 @@
 // We need to specify some content-type mapping, so the resources get delivered with the
 // right content type and are displayed correctly in the browser
 char contentTypes[][2][32] = {
-  {".txt", "text/plain"},
-  {".png",  "image/png"},
-  {".jpg",  "image/jpg"},
+  {".txt",  "text/plain"},
+  {".html",  "text/html"},
+  {".png",   "image/png"},
+  {".jpg",   "image/jpg"},
   {"", ""}
 };
 
@@ -63,18 +64,19 @@ SSLCert cert = SSLCert(
 HTTPSServer secureServer = HTTPSServer(&cert);
 
 // Declare some handler functions for the various URLs on the server
-// The signature is always the same for those functions. They get two parameters,
-// which are pointers to the request data (read request body, headers, ...) and
-// to the response data (write response, set status code, ...)
+// See the static-page example for how handler functions work.
+// The comments in setup() describe what each handler function does in this example.
 void handleRoot(HTTPRequest * req, HTTPResponse * res);
 void handleFormUpload(HTTPRequest * req, HTTPResponse * res);
 void handleFormEdit(HTTPRequest * req, HTTPResponse * res);
 void handleFile(HTTPRequest * req, HTTPResponse * res);
 void handleDirectory(HTTPRequest * req, HTTPResponse * res);
 void handle404(HTTPRequest * req, HTTPResponse * res);
 
+// As we have a file editor where the content of a file is pasted into a <textarea>,
+// we need to take care of encoding special characters correctly.
 std::string htmlEncode(std::string data) {
-  // Quick and dirty: doesn't handle control chars and such.
+  // Quick and dirty: doesn't handle control chars and such. Don't use it in production code
   const char *p = data.c_str();
   std::string rv = "";
   while(p && *p) {
@@ -110,26 +112,37 @@ void setup() {
 
   // For every resource available on the server, we need to create a ResourceNode
   // The ResourceNode links URL and HTTP method to a handler function
+
+  // The root node shows a static page with a link to the file directory and a small
+  // HTML form that allows uploading new forms
   ResourceNode * nodeRoot = new ResourceNode("/", "GET", &handleRoot);
+  // The handleFormUpload handler handles the file upload from the root node. As the form
+  // is submitted via post, we need to specify that as handler method here:
   ResourceNode * nodeFormUpload = new ResourceNode("/upload", "POST", &handleFormUpload);
+
+  // For the editor, we use the same handler function and register it with the GET and POST
+  // method. The handler decides what to do based on the method used to call it:
   ResourceNode * nodeFormEdit = new ResourceNode("/edit", "GET", &handleFormEdit);
   ResourceNode * nodeFormEditDone = new ResourceNode("/edit", "POST", &handleFormEdit);
+
+  // To keep track of all uploaded files, we provide a directory listing here with an edit
+  // button for text-based files:
   ResourceNode * nodeDirectory = new ResourceNode("/public", "GET", &handleDirectory);
+
+  // And of course we need some way to retrieve the file again. We use the placeholder
+  // feature in the path to do so:
   ResourceNode * nodeFile = new ResourceNode("/public/*", "GET", &handleFile);
 
   // 404 node has no URL as it is used for all requests that don't match anything else
   ResourceNode * node404  = new ResourceNode("", "GET", &handle404);
 
-  // Add the root nodes to the server
+  // Add all nodes to the server so they become accessible:
   secureServer.registerNode(nodeRoot);
   secureServer.registerNode(nodeFormUpload);
   secureServer.registerNode(nodeFormEdit);
   secureServer.registerNode(nodeFormEditDone);
   secureServer.registerNode(nodeDirectory);
   secureServer.registerNode(nodeFile);
-
-  // Add the 404 not found node to the server.
-  // The path is ignored for the default node.
   secureServer.setDefaultNode(node404);
 
   Serial.println("Starting server...");
@@ -152,22 +165,28 @@ void handleRoot(HTTPRequest * req, HTTPResponse * res) {
   // We want to deliver a simple HTML page, so we send a corresponding content type:
   res->setHeader("Content-Type", "text/html");
 
-  // The response implements the Print interface, so you can use it just like
-  // you would write to Serial etc.
+  // Just the regular HTML document structure, nothing special to forms here....
   res->println("<!DOCTYPE html>");
   res->println("<html>");
   res->println("<head><title>Very simple file server</title></head>");
   res->println("<body>");
   res->println("<h1>Very simple file server</h1>");
   res->println("<p>This is a very simple file server to demonstrate the use of POST forms. </p>");
+
+  // The link to the file listing (/public is produced by handleDirectory())
   res->println("<h2>List existing files</h2>");
   res->println("<p>See <a href=\"/public\">/public</a> to list existing files and retrieve or edit them.</p>");
+
+  // Here comes the upload form. Note the enctype="multipart/form-data". Only by setting that enctype, you
+  // will be able to upload a file. If you miss it, the file field will only contain the filename.
+  // Method is POST, which matches the way that nodeFormUpload is configured in setup().
   res->println("<h2>Upload new file</h2>");
   res->println("<p>This form allows you to upload files (text, jpg and png supported best). It demonstrates multipart/form-data.</p>");
   res->println("<form method=\"POST\" action=\"/upload\" enctype=\"multipart/form-data\">");
   res->println("file: <input type=\"file\" name=\"file\"><br>");
   res->println("<input type=\"submit\" value=\"Upload\">");
   res->println("</form>");
+
   res->println("</body>");
   res->println("</html>");
 }
@@ -180,34 +199,66 @@ void handleFormUpload(HTTPRequest * req, HTTPResponse * res) {
   // to be multipart/form-data.
   HTTPBodyParser *parser;
   std::string contentType = req->getHeader("Content-Type");
+
+  // The content type may have additional properties after a semicolon, for exampel:
+  // Content-Type: text/html;charset=utf-8
+  // Content-Type: multipart/form-data;boundary=------s0m3w31rdch4r4c73rs
+  // As we're interested only in the actual mime _type_, we strip everything after the
+  // first semicolon, if one exists:
   size_t semicolonPos = contentType.find(";");
   if (semicolonPos != std::string::npos) {
     contentType = contentType.substr(0, semicolonPos);
   }
+
+  // Now, we can decide based on the content type:
   if (contentType == "multipart/form-data") {
     parser = new HTTPMultipartBodyParser(req);
   } else {
     Serial.printf("Unknown POST Content-Type: %s\n", contentType.c_str());
     return;
   }
-  // We iterate over the fields. Any field with a filename is uploaded
+
   res->println("<html><head><title>File Upload</title></head><body><h1>File Upload</h1>");
+
+  // We iterate over the fields. Any field with a filename is uploaded.
+  // Note that the BodyParser consumes the request body, meaning that you can iterate over the request's
+  // fields only a single time. The reason for this is that it allows you to handle large requests
+  // which would not fit into memory.
   bool didwrite = false;
+
+  // parser->nextField() will move the parser to the next field in the request body (field meaning a
+  // form field, if you take the HTML perspective). After the last field has been processed, nextField()
+  // returns false and the while loop ends.
   while(parser->nextField()) {
+    // For Multipart data, each field has three properties:
+    // The name ("name" value of the <input> tag)
+    // The filename (If it was a <input type="file">, this is the filename on the machine of the
+    //   user uploading it)
+    // The mime type (It is determined by the client. So do not trust this value and blindly start
+    //   parsing files only if the type matches)
     std::string name = parser->getFieldName();
     std::string filename = parser->getFieldFilename();
     std::string mimeType = parser->getFieldMimeType();
+    // We log all three values, so that you can observe the upload on the serial monitor:
     Serial.printf("handleFormUpload: field name='%s', filename='%s', mimetype='%s'\n", name.c_str(), filename.c_str(), mimeType.c_str());
+
     // Double check that it is what we expect
     if (name != "file") {
       Serial.println("Skipping unexpected field");
       break;
     }
-    // Should check file name validity and all that, but we skip that.
+
+    // You should check file name validity and all that, but we skip that to make the core
+    // concepts of the body parser functionality easier to understand.
     std::string pathname = "/public/" + filename;
+
+    // Create a new file on spiffs to stream the data into
     File file = SPIFFS.open(pathname.c_str(), "w");
     size_t fileLength = 0;
     didwrite = true;
+
+    // With endOfField you can check whether the end of field has been reached or if there's
+    // still data pending. With multipart bodies, you cannot know the field size in advance.
     while (!parser->endOfField()) {
       byte buf[512];
       size_t readLength = parser->read(buf, 512);
@@ -225,54 +276,107 @@ void handleFormUpload(HTTPRequest * req, HTTPResponse * res) {
 }
 
 void handleFormEdit(HTTPRequest * req, HTTPResponse * res) {
+  // This handler function does two things:
+  // For GET: Show an editor
+  // For POST: Handle editor submit
   if (req->getMethod() == "GET") {
     // Initial request. Get filename from request parameters and return form.
+    // The filename is in the URL, so we need to use the query params here:
+    // (URL is like /edit?filename=something.txt)
     auto params = req->getParams();
     std::string filename;
     bool hasFilename = params->getQueryParameter("filename", filename);
     std::string pathname = std::string("/public/") + filename;
+
+    // Start writing the HTML output
     res->println("<html><head><title>Edit File</title><head><body>");
+
+    // Try to open the file from SPIFFS
     File file = SPIFFS.open(pathname.c_str());
     if (!hasFilename) {
+      // No ?filename=something parameter was given
       res->println("<p>No filename specified.</p>");
+
     } else if (!file.available()) {
+      // The file didn't exist in the SPIFFS
       res->printf("<p>File not found: %s</p>\n", pathname.c_str());
+
     } else {
+      // We have a file, render the form:
       res->printf("<h2>Edit content of %s</h2>\n", pathname.c_str());
+
+      // Start writing the form. The file content will be shown in a <textarea>, so there is
+      // no file upload happening (from the HTML perspective). For that reason, we use the
+      // x-www-form-urlencoded enctype as it is much more efficient:
       res->println("<form method=\"POST\" enctype=\"application/x-www-form-urlencoded\">");
+
+      // Store the filename hidden in the form so that we know which file to update when the form
+      // is submitted
       res->printf("<input name=\"filename\" type=\"hidden\" value=\"%s\">", filename.c_str());
       res->print("<textarea name=\"content\" rows=\"24\" cols=\"80\">");
-      // Read the file and write it to the response
+      
+      // Read the file from SPIFFS and write it to the HTTP response body
       size_t length = 0;
       do {
         char buffer[256];
         length = file.read((uint8_t *)buffer, 256);
         std::string bufferString(buffer, length);
+        // htmlEncode handles conversions of < to &lt; so that the form is rendered correctly
         bufferString = htmlEncode(bufferString);
         res->write((uint8_t *)bufferString.c_str(), bufferString.size());
       } while (length > 0);
+
+      // Finalize the form with a submitt button
       res->println("</textarea><br>");
       res->println("<input type=\"submit\" value=\"Save\">");
       res->println("</form>");
     }
     res->println("</body></html>");
+
   } else { // method != GET
     // Assume POST request. Contains submitted data.
     res->println("<html><head><title>File Edited</title><head><body><h1>File Edited</h1>");
+
+    // The form is submitted with the x-www-form-urlencoded content type, so we need the
+    // HTTPURLEncodedBodyParser to read the fields.
+    // Note that the content of the file's content comes from a <textarea>, so we
+    // can use the URL encoding here, since no file upload from an <input type="file"
+    // is involved.
     HTTPURLEncodedBodyParser parser(req);
+
+    // The bodyparser will consume the request body. That means you can iterate over the
+    // fields only ones. For that reason, we need to create variables for all fields that
+    // we expect. So when parsing is done, you can process the field values from your
+    // temporary variables.
     std::string filename;
     bool savedFile = false;
+    
+    // Iterate over the fields from the request body by calling nextField(). This function
+    // will update the field name and value of the body parsers. If the last field has been
+    // reached, it will return false and the while loop stops.
     while(parser.nextField()) {
+      // Get the field name, so that we can decide what the value is for
       std::string name = parser.getFieldName();
+
       if (name == "filename") {
+        // Read the filename from the field's value, add the /public prefix and store it in
+        // the filename variable.
         char buf[512];
         size_t readLength = parser.read((byte *)buf, 512);
         filename = std::string("/public/") + std::string(buf, readLength);
+
       } else if (name == "content") {
+        // Browsers must return the fields in the order that they are placed in
+        // the HTML form, so if the broweser behaves correctly, this condition will
+        // never be true. We include it for safety reasons.
         if (filename == "") {
           res->println("<p>Error: form contained content before filename.</p>");
           break;
         }
+
+        // With parser.read() and parser.endOfField(), we can stream the field content
+        // into a buffer. That allows handling arbitrarily-sized field contents. Here,
+        // we use it and write the file contents directly to the SPIFFS:
         size_t fieldLength = 0;
         File file = SPIFFS.open(filename.c_str(), "w");
         savedFile = true;
@@ -284,6 +388,7 @@ void handleFormEdit(HTTPRequest * req, HTTPResponse * res) {
         }
         file.close();
         res->printf("<p>Saved %d bytes to %s</p>", int(fieldLength), filename.c_str());
+
       } else {
         res->printf("<p>Unexpected field %s</p>", name.c_str());
       }
@@ -297,7 +402,10 @@ void handleFormEdit(HTTPRequest * req, HTTPResponse * res) {
 
 void handleDirectory(HTTPRequest * req, HTTPResponse * res) {
   res->println("<html><head><title>File Listing</title><head><body>");
+
+  // We read the SPIFFS folder public and render all files to the HTML page:
   File d = SPIFFS.open("/public");
+
   if (!d.isDirectory()) {
     res->println("<p>No files found.</p>");
   } else {
@@ -306,8 +414,13 @@ void handleDirectory(HTTPRequest * req, HTTPResponse * res) {
     File f = d.openNextFile();
     while (f) {
       std::string pathname(f.name());
+
+      // We render a link to /public/... for each file that we find
       res->printf("<li><a href=\"%s\">%s</a>", pathname.c_str(), pathname.c_str());
+
       if (pathname.rfind(".txt") != std::string::npos) {
+        // And if the file is a text file, we also include an editor link like
+        // /edit?filename=... to open the editor, which is created by handleFormEdit.
         std::string filename = pathname.substr(8); // Remove /public/
         res->printf(" <a href=\"/edit?filename=%s\">[edit]</a>", filename.c_str());
       }