Expression Engine 2 the Upload Path Does Not Appear to Be Valid.
Introduction
At one time or some other when building our Node application nosotros have been faced with uploading a photo (usually from a form) to be used as a profile photograph for a user in our app. In addition, we usually accept to store the photo in the local filesystem (during evolution) or fifty-fifty in the cloud for easy admission. Since this is a very common task, there are lots of tools available which we can leverage to handle the individual parts of the process.
In this tutorial, we will come across how to upload a photo and manipulate it (resize, crop, greyscale, etc) earlier writing it to storage. Nosotros will limit ourselves to storing files in the local filesystem for simplicity.
Prerequisites
Nosotros will exist using the following packages to build our application:
- express: A very popular Node server.
- lodash: A very popular JavaScript library with lots of utility functions for working with arrays, strings, objects and functional programming.
- multer: A parcel for extracting files from
multipart/form-datarequests. - jimp: An image manipulation package.
- dotenv: A package for calculation
.envvariables toprocess.env. - mkdirp: A packet for creating nested directory structure.
- concat-stream: A package for creating a writable stream that concatenates all the data from a stream and calls a callback with the result.
- streamifier: A parcel to convert a Buffer/String into a readable stream.
Project Goals
We desire to take over the uploaded file stream from Multer and so manipulate the stream buffer (paradigm) withal nosotros wish using Jimp, before writing the image to storage (local filesystem). This will crave us to create a custom storage engine to utilize with Multer — which we will be doing in this tutorial.
Here is the terminate result of what we will be building in this tutorial:
Stride 1 — Getting Started
We will brainstorm past creating a new Express app using the Express generator. If you don't have the Express generator already y'all volition demand to install it first by running the following command on your control line terminal:
- npm install express-generator -g
In one case you have the Express generator, you can at present run the following commands to create a new Express app and install the dependencies for Express. We will be using ejs as our view engine:
- limited --view=ejs photo-uploader-app
- cd photo-uploader-app
- npm install
Adjacent, we will install the remaining dependencies nosotros need for our projection:
- npm install --salve lodash multer jimp dotenv concat-stream streamifier mkdirp
Footstep 2 — Configuring the Basics
Earlier we keep, our app will need some form configuration. We will create a .env file on our project root directory and add some environment variables. The .env file should look like the following snippet.
AVATAR_FIELD = avatar AVATAR_BASE_URL = /uploads/avatars AVATAR_STORAGE = uploads/avatars Next, we will load our environment variables into procedure.env using dotenv so that we can access them in our app. To practice this, nosotros volition add together the following line to the app.js file. Ensure you add this line at the point where you are loading the dependencies. It must come before all road imports and before creating the Express app instance.
app.js
var dotenv = crave ( 'dotenv' ) . config ( ) ; Now nosotros can access our environment variables using procedure.env. For example: process.env.AVATAR_STORAGE should contain the value uploads/avatars. Nosotros will become ahead to edit our index route file routes/index.js to add some local variables we will exist needing in our view. We volition add two local variables:
- title: The title of our index page:
Upload Avatar - avatar_field: The name of the input field for our avatar photograph. We will be getting this from
procedure.env.AVATAR_FIELD
Modify the GET / route equally follows:
routes/index.js
router. get ( '/' , function ( req, res, adjacent ) { res. render ( 'alphabetize' , { championship : 'Upload Avatar' , avatar_field : process.env. AVATAR_FIELD } ) ; } ) ; Footstep 3 — Preparing the View
Allow's begin by creating the basic markup for our photograph upload course by modifying the views/index.ejs file. For the sake of simplicity we will add the styles direct on our view just to requite information technology a slightly prissy expect. Run across the following code for the markup of our page.
views/index.ejs
<html class = "no-js" > <caput > <meta charset = "utf-viii" > <meta http-equiv = "X-UA-Uniform" content = "IE=edge" > <meta name = "viewport" content = "width=device-width, initial-calibration=1" > <title > <%= title %> </title > <style type = "text/css" > * { font : 600 16px system-ui, sans-serif; } form { width : 320px; margin : 50px auto; text-align : eye; } grade > fable { font-size : 36px; color : #3c5b6d; padding : 150px 0 20px; } form > input[type=file], grade > input[type=file]:earlier { display : cake; width : 240px; tiptop : 50px; margin : 0 motorcar; line-acme : 50px; text-align : eye; cursor : pointer; } form > input[blazon=file] { position : relative; } class > input[type=file]:earlier { content : 'Choose a Photograph' ; position : absolute; peak : -2px; left : -2px; color : #3c5b6d; font-size : 18px; background : #fff; border-radius : 3px; border : 2px solid #3c5b6d; } form > push button[type=submit] { border-radius : 3px; font-size : 18px; display : block; edge : none; color : #fff; cursor : arrow; background : #2a76cd; width : 240px; margin : 20px auto; padding : 15px 20px; } </style > </caput > <trunk > <form action = "/upload" method = "POST" enctype = "multipart/form-data" > <legend > Upload Avatar </legend > <input blazon = "file" name = "<%= avatar_field %>" > <push button type = "submit" grade = "btn btn-principal" > Upload </push button > </grade > </body > </html > Observe how we accept used our local variables on our view to ready the title and the proper name of the avatar input field. You lot will notice that we are using enctype="multipart/form-data" on our form since we will exist uploading a file. You will besides see that nosotros accept set the form to brand a Mail request to the /upload route (nosotros will implement later) on submission.
Now permit's outset the app for the first time using npm showtime.
- npm outset
If you accept been post-obit correctly everything should run without errors. Just visit localhost:3000 on your browser. The page should look like the following screenshot:
Stride four — Creating the Multer Storage Engine
So far, trying to upload a photo through our class will consequence in an error because we've not created the handler for the upload request. We are going to implement the /upload road to actually handle the upload and we will be using the Multer package for that. If y'all are not already familiar with the Multer package yous can bank check the Multer package on Github.
We will accept to create a custom storage engine to utilize with Multer. Let's create a new folder in our project root named helpers and create a new file AvatarStorage.js inside information technology for our custom storage engine. The file should contain the post-obit design code snippet:
helpers/AvatarStorage.js
// Load dependencies var _ = crave ( 'lodash' ) ; var fs = require ( 'fs' ) ; var path = require ( 'path' ) ; var Jimp = require ( 'jimp' ) ; var crypto = crave ( 'crypto' ) ; var mkdirp = require ( 'mkdirp' ) ; var concat = require ( 'concat-stream' ) ; var streamifier = require ( 'streamifier' ) ; // Configure UPLOAD_PATH // process.env.AVATAR_STORAGE contains uploads/avatars var UPLOAD_PATH = path. resolve (__dirname, '..' , process.env. AVATAR_STORAGE ) ; // create a multer storage engine var AvatarStorage = function ( options ) { // this serves every bit a constructor function AvatarStorage ( opts ) { } // this generates a random cryptographic filename AvatarStorage .epitome. _generateRandomFilename = part ( ) { } // this creates a Writable stream for a filepath AvatarStorage .prototype. _createOutputStream = function ( filepath, cb ) { } // this processes the Jimp paradigm buffer AvatarStorage .prototype. _processImage = part ( image, cb ) { } // multer requires this for handling the uploaded file AvatarStorage .prototype. _handleFile = function ( req, file, cb ) { } // multer requires this for destroying file AvatarStorage .epitome. _removeFile = part ( req, file, cb ) { } // create a new instance with the passed options and return it return new AvatarStorage (options) ; } ; // export the storage engine module.exports = AvatarStorage; Allow's begin to add together the implementations for the listed functions in our storage engine. Nosotros will begin with the constructor office.
// this serves every bit a constructor function AvatarStorage ( opts ) { var baseUrl = process.env. AVATAR_BASE_URL ; var allowedStorageSystems = [ 'local' ] ; var allowedOutputFormats = [ 'jpg' , 'png' ] ; // fallback for the options var defaultOptions = { storage : 'local' , output : 'png' , greyscale : false , quality : 70 , square : truthful , threshold : 500 , responsive : false , } ; // extend default options with passed options var options = (opts && _. isObject (opts) ) ? _. selection (opts, _. keys (defaultOptions) ) : { } ; options = _. extend (defaultOptions, options) ; // check the options for correct values and use fallback value where necessary this .options = _. forIn (options, function ( value, primal, object ) { switch (cardinal) { case 'square' : case 'greyscale' : case 'responsive' : object[cardinal] = _. isBoolean (value) ? value : defaultOptions[key] ; break ; case 'storage' : value = String (value) . toLowerCase ( ) ; object[key] = _. includes (allowedStorageSystems, value) ? value : defaultOptions[central] ; suspension ; case 'output' : value = String (value) . toLowerCase ( ) ; object[fundamental] = _. includes (allowedOutputFormats, value) ? value : defaultOptions[key] ; break ; case 'quality' : value = _. isFinite (value) ? value : Number (value) ; object[key] = (value && value >= 0 && value <= 100 ) ? value : defaultOptions[key] ; break ; case 'threshold' : value = _. isFinite (value) ? value : Number (value) ; object[key] = (value && value >= 0 ) ? value : defaultOptions[central] ; suspension ; } } ) ; // set the upload path this .uploadPath = this .options.responsive ? path. bring together ( UPLOAD_PATH , 'responsive' ) : UPLOAD_PATH ; // set up the upload base of operations url this .uploadBaseUrl = this .options.responsive ? path. bring together (baseUrl, 'responsive' ) : baseUrl; if ( this .options.storage == 'local' ) { // if upload path does not be, create the upload path structure !fs. existsSync ( this .uploadPath) && mkdirp. sync ( this .uploadPath) ; } } Here, we divers our constructor function to accept a couple of options. We besides added some default (fallback) values for these options in case they are non provided or they are invalid. You can tweak this to comprise more than options depending on what you lot want, but for this tutorial nosotros volition stick with the post-obit options for our storage engine.
- storage: The storage filesystem. Simply allowed value is
'local'for local filesystem. Defaults to'local'. You can implement other storage filesystems (likeAmazon S3) if you wish. - output: The epitome output format. Can exist
'jpg'or'png'. Defaults to'png'. - greyscale: If set to
true, the output image will be greyscale. Defaults tosimulated. - quality: A number betwixt 0: 100 that determines the quality of the output image. Defaults to
70. - square: If prepare to
true, the prototype volition be cropped to a square. Defaults tofalse. - threshold: A number that restricts the smallest dimension (in
px) of the output image. The default value is500. If the smallest dimension of the paradigm exceeds this number, the image is resized and then that the smallest dimension is equal to the threshold. - responsive: If ready to
true, three output images of different sizes (lg,mdandsm) will be created and stored in their respective folders. Defaults tofalse.
Let'due south implement the methods for creating the random filenames and the output stream for writing to the files:
// this generates a random cryptographic filename AvatarStorage .epitome. _generateRandomFilename = role ( ) { // create pseudo random bytes var bytes = crypto. pseudoRandomBytes ( 32 ) ; // create the md5 hash of the random bytes var checksum = crypto. createHash ( 'MD5' ) . update (bytes) . assimilate ( 'hex' ) ; // return equally filename the hash with the output extension return checksum + '.' + this .options.output; } ; // this creates a Writable stream for a filepath AvatarStorage .epitome. _createOutputStream = role ( filepath, cb ) { // create a reference for this to use in local functions var that = this ; // create a writable stream from the filepath var output = fs. createWriteStream (filepath) ; // set callback fn equally handler for the error outcome output. on ( 'error' , cb) ; // set handler for the finish event output. on ( 'cease' , role ( ) { cb ( null , { destination : that.uploadPath, baseUrl : that.uploadBaseUrl, filename : path. basename (filepath) , storage : that.options.storage } ) ; } ) ; // return the output stream return output; } ; Hither, we apply crypto to create a random md5 hash to apply as filename and appended the output from the options every bit the file extension. We also defined our helper method to create writable stream from the given filepath and then render the stream. Detect that a callback function is required, since we are using it on the stream event handlers.
Adjacent we volition implement the _processImage() method that does the actual image processing. Here is the implementation:
// this processes the Jimp epitome buffer AvatarStorage .image. _processImage = function ( image, cb ) { // create a reference for this to utilise in local functions var that = this ; var batch = [ ] ; // the responsive sizes var sizes = [ 'lg' , 'md' , 'sm' ] ; var filename = this . _generateRandomFilename ( ) ; var mime = Jimp. MIME_PNG ; // create a clone of the Jimp image var clone = image. clone ( ) ; // fetch the Jimp image dimensions var width = clone.bitmap.width; var height = clone.bitmap.height; var square = Math. min (width, height) ; var threshold = this .options.threshold; // resolve the Jimp output mime type switch ( this .options.output) { case 'jpg' : mime = Jimp. MIME_JPEG ; break ; example 'png' : default : mime = Jimp. MIME_PNG ; pause ; } // auto scale the paradigm dimensions to fit the threshold requirement if (threshold && square > threshold) { clone = (foursquare == width) ? clone. resize (threshold, Jimp. AUTO ) : clone. resize (Jimp. Automobile , threshold) ; } // crop the image to a square if enabled if ( this .options.foursquare) { if (threshold) { square = Math. min (square, threshold) ; } // fetch the new image dimensions and crop clone = clone. crop ( (clone.bitmap.width: square) / 2 , (clone.bitmap.elevation: square) / 2 , square, foursquare) ; } // catechumen the image to greyscale if enabled if ( this .options.greyscale) { clone = clone. greyscale ( ) ; } // set the epitome output quality clone = clone. quality ( this .options.quality) ; if ( this .options.responsive) { // map through the responsive sizes and push them to the batch batch = _. map (sizes, function ( size ) { var outputStream; var image = cipher ; var filepath = filename. split ( '.' ) ; // create the consummate filepath and create a writable stream for it filepath = filepath[ 0 ] + '_' + size + '.' + filepath[ one ] ; filepath = path. join (that.uploadPath, filepath) ; outputStream = that. _createOutputStream (filepath, cb) ; // scale the image based on the size switch (size) { case 'sm' : image = clone. clone ( ) . calibration ( 0.3 ) ; break ; example 'physician' : image = clone. clone ( ) . scale ( 0.seven ) ; interruption ; instance 'lg' : prototype = clone. clone ( ) ; break ; } // return an object of the stream and the Jimp image return { stream : outputStream, image : image } ; } ) ; } else { // button an object of the writable stream and Jimp image to the batch batch. push button ( { stream : that. _createOutputStream (path. join (that.uploadPath, filename) , cb) , paradigm : clone } ) ; } // process the batch sequence _. each (batch, part ( current ) { // get the buffer of the Jimp image using the output mime blazon electric current.prototype. getBuffer (mime, function ( err, buffer ) { if (that.options.storage == 'local' ) { // create a read stream from the buffer and pipe it to the output stream streamifier. createReadStream (buffer) . pipe (electric current.stream) ; } } ) ; } ) ; } ; A lot is going on in this method but here is a summary of what it is doing:
- Generates a random filename, resolves the Jimp output paradigm mime type and gets the paradigm dimensions.
- Resize the paradigm if required, based on the threshold requirements to ensure that the smallest dimension does not exceed the threshold.
- Crop the prototype to a square if enabled in the options.
- Convert the image to greyscale if enabled in the options.
- Set the epitome output quality from the options.
- If responsive is enabled, the paradigm is cloned and scaled for each of the responsive sizes (
lg,medicoandsm) and and so an output stream is created using the_createOutputStream()method for each image file of the respective sizes. The filename for each size takes the format[random_filename_hash]_[size].[output_extension]. Then the image clone and the stream are put in a batch for processing. - If responsive is disabled, then only the current prototype and an output stream for information technology is put in a batch for processing.
- Finally, each item in the batch is processed by converting the Jimp image buffer into a readable stream using streamifier and then piping the readable stream to the output stream.
Now we volition implement the remaining methods and we will exist done with our storage engine.
// multer requires this for handling the uploaded file AvatarStorage .prototype. _handleFile = role ( req, file, cb ) { // create a reference for this to utilize in local functions var that = this ; // create a writable stream using concat-stream that will // concatenate all the buffers written to it and laissez passer the // complete buffer to a callback fn var fileManipulate = concat ( office ( imageData ) { // read the image buffer with Jimp // it returns a promise Jimp. read (imageData) . then ( function ( image ) { // process the Jimp paradigm buffer that. _processImage (image, cb) ; } ) . take hold of (cb) ; } ) ; // write the uploaded file buffer to the fileManipulate stream file.stream. pipe (fileManipulate) ; } ; // multer requires this for destroying file AvatarStorage .prototype. _removeFile = function ( req, file, cb ) { var matches, pathsplit; var filename = file.filename; var _path = path. join ( this .uploadPath, filename) ; var paths = [ ] ; // delete the file properties delete file.filename; delete file.destination; delete file.baseUrl; delete file.storage; // create paths for responsive images if ( this .options.responsive) { pathsplit = _path. split ( '/' ) ; matches = pathsplit. pop ( ) . match ( / ^(.+?)_.+?\.(.+)$ / i ) ; if (matches) { paths = _. map ( [ 'lg' , 'md' , 'sm' ] , function ( size ) { return pathsplit. join ( '/' ) + '/' + (matches[ ane ] + '_' + size + '.' + matches[ 2 ] ) ; } ) ; } } else { paths = [_path] ; } // delete the files from the filesystem _. each (paths, function ( _path ) { fs. unlink (_path, cb) ; } ) ; } ; Our storage engine is now set up for use with Multer.
Pace five — Implementing the POST /upload Road
Before we define the route, nosotros volition need to setup Multer for use in our road. Permit's go ahead to edit the routes/index.js file to add together the post-obit:
routes/index.js
var express = crave ( 'express' ) ; var router = express. Router ( ) ; /** * CODE ADDITION * * The following code is added to import boosted dependencies * and setup Multer for use with the /upload route. */ // import multer and the AvatarStorage engine var _ = require ( 'lodash' ) ; var path = require ( 'path' ) ; var multer = require ( 'multer' ) ; var AvatarStorage = crave ( '../helpers/AvatarStorage' ) ; // setup a new case of the AvatarStorage engine var storage = AvatarStorage ( { square : true , responsive : true , greyscale : true , quality : ninety } ) ; var limits = { files : 1 , // allow simply 1 file per asking fileSize : 1024 * 1024 , // 1 MB (max file size) } ; var fileFilter = function ( req, file, cb ) { // supported image file mimetypes var allowedMimes = [ 'paradigm/jpeg' , 'image/pjpeg' , 'image/png' , 'image/gif' ] ; if (_. includes (allowedMimes, file.mimetype) ) { // allow supported image files cb ( null , truthful ) ; } else { // throw mistake for invalid files cb ( new Error ( 'Invalid file type. But jpg, png and gif prototype files are immune.' ) ) ; } } ; // setup multer var upload = multer ( { storage : storage, limits : limits, fileFilter : fileFilter } ) ; /* Code Improver ENDS HERE */ Hither, we are enabling square cropping, responsive images and setting the threshold for our storage engine. We also add limits to our Multer configuration to ensure that the maximum file size is 1 MB and to ensure that non-image files are not uploaded.
Now permit's add the POST /upload route every bit follows:
/* routes/index.js */ /** * Lawmaking Add-on * * The following code is added to configure the POST /upload route * to upload files using the already defined Multer configuration */ router. postal service ( '/upload' , upload. single (process.env. AVATAR_FIELD ) , function ( req, res, next ) { var files; var file = req.file.filename; var matches = file. match ( / ^(.+?)_.+?\.(.+)$ / i ) ; if (matches) { files = _. map ( [ 'lg' , 'doctor' , 'sm' ] , role ( size ) { render matches[ 1 ] + '_' + size + '.' + matches[ two ] ; } ) ; } else { files = [file] ; } files = _. map (files, part ( file ) { var port = req.app. become ( 'port' ) ; var base of operations = req.protocol + '://' + req.hostname + (port ? ':' + port : '' ) ; var url = path. bring together (req.file.baseUrl, file) . replace ( / [\\\/]+ / 1000 , '/' ) . replace ( / ^[\/]+ / g , '' ) ; return (req.file.storage == 'local' ? base : '' ) + '/' + url; } ) ; res. json ( { images : files } ) ; } ) ; /* Lawmaking Improver ENDS Here */ Notice how we passed the Multer upload middleware before our route handler. The single() method allows us to upload only one file that will be stored in req.file. It takes every bit get-go parameter, the proper name of the file input field which we access from procedure.env.AVATAR_FIELD.
Now let's first the app over again using npm outset.
- npm commencement
visit localhost:3000 on your browser and try to upload a photo. Here is a sample screenshot I got from testing the upload road on Postman using our current configuration options:
You can tweak the configuration options of the storage engine in our Multer setup to get different results.
Determination
In this tutorial, we have been able to create a custom storage engine for utilise with Multer which manipulates uploaded images using Jimp and then writes them to storage. For a complete code sample of this tutorial, checkout the advanced-multer-node-sourcecode repository on Github.
fewingsthavatabot.blogspot.com
Source: https://www.digitalocean.com/community/tutorials/how-to-add-advanced-photo-uploads-in-node-and-express