Monthly Archives: December 2012

File Chunking/Partitioning is now available in 3.2

The newest feature added to Fine Uploader 3.2 is the ability to split large files into smaller chunks and spread them out over as many requests. This has many advantages, but primarily enhances the auto/manual retry feature and sets the stage for an upcoming feature that will allow users to automatically resume an uncompleted upload from a previous session. Chunking also allows you to overcome the hardcoded request limits enforced by most browsers. For example, Chrome and Firefox limit a file upload request size to about 4 GB. By uploading the file in chunks, you can overcome this.

Background

File chunking is made possible by the File API and, more specifically, the slice function on the File object prototype, inherited from the Blob object prototype. This function allows us to cut up a file into parts. Each part is a Blob and is then sent with the XHR request. Currently, file chunking is possible in Chrome, Firefox, Safari in IOS6, Safari in OS X, and Internet Explorer 10. Note that chunking is disabled in all Android browsers, as the Blob.slice implementation is broken in Android.

Callbacks

I have also provided a new callback: onUploadChunk. This is invoked before each chunk is sent. The file ID, name, and some metadata specific to the chunk are all passed into the callback. You can make use of the setParams function on this callback (and all other callbacks) to make any adjustments to the parameters sent along with this chunk. Please see the callbacks section of the readme for more details on this callback.

Server-side Examples

I have updated the Java example to handle file chunking. Andrew Valums has updated the PHP example to handle chunking as well. These changes will address chunking with multipart encoded requests. Hopefully, other server-side examples will be updated in the near future by contributors who are more familiar with the other server-side languages represented in the server directory.

Server-side handling

If you are a potential contributor who would like to modify an existing server-side example to handle chunked files, or if you are simply a user of Fine Uploader who would like to make use of this new feature in your server-side language of choice, I’ll do my best to explain how to best handle these requests in a language-agnostic manner.

Each chunk is sent in order and the response is checked to determine success. If the response from the server indicates failure, the uploader will attempt to retry sending the file starting with the last failed chunk (assuming either auto or manual retry has been enabled). Various parameters are sent by Fine Uploader along with each chunked request. Please see the chunking.paramNames option documentation in the readme for more details about these parameters.

Generally speaking, you can determine if a request refers to a file chunk by checking for the presence of a “qqpartindex” parameter. Other parameters standard with every chunked file request are “qqpartbyteoffset”, “qqchunksize”, “qqtotalfilesize”, and “qqtotalparts”. Note that the “qqtotalfilesize” parameter is also sent with ALL multipart-encoded requests that are initiated by the XHR uploader. Also, the “qqfilename” parameter is only sent along with chunked file requests that are multipart encoded. This parameter is important in this context since the filename reported in the Content-Disposition header has a value of “blob”, or an empty string when a Blob is included in the request. So, in order to determine the original filename when dealing with a MPE request, you must read the “qqfilename” parameter.

Each chunk will be sent in a separate request. After processing the request, you should save the chunk in a temporary location. To avoid collisions with other chunks, you should name the chunks after the “qquuid” parameter value. Note that this is a version 4 UUID, but should be unique enough for most, if not all applications. If you are especially worried about collisions between UUIDs, you can combine the UUID with the original file name (“qqfilename”) to decrease the chance of collisions further. Each temporary chunk file should be also be named with the chunk index. After all chunks have been received for a file, you should then combine all chunks to arrive at the original file. This is typically done as part of handling the final chunked request. If, for some reason, you have lost one of the previous chunks, or one of the previous chunks is no longer valid, you can return a “reset” JSON property with a value of “true” in your response. When Fine Uploader receives such a property in the chunked request response, it will fail the upload and then restart the upload with the first chunk again on the next attempt (assuming auto or manual retry has been enabled).

You may use the java example in the server folder as a reference.

Simple Client-Side Example

Here is an example of a typical, suggested setup (client-side) if file chunking is a desired feature:

var uploader = new qq.FineUploader({
    element: document.getElementById('myUploader'),
    request: {
        endpoint: '/my/endpoint'
    },
    retry: {
        enableAuto: true
    },
    chunking: {
        enabled: true
    }
});

The next planned feature is to allow users to resume a previously stopped or failed upload.

-Ray Nicholus

Fine Uploader 3.1

Fine Uploader 3.1 has been released, and I have done my best to include a bunch of improvements, along with some new features.

As always, please see the downloads page to download the library.

Known Issues

  • #539 – Uploads broken in Android 2.3 and older. This is fixed in the 3.1.1 hotfix release.

Features & Enhancements

  • #518 – Allow parameters to be specified for a specific file, and at any time during the upload process. This affects the setParams function. I have explained how this works, in detail, in a blog post.
  • #510 – Allow additional input elements and File objects to be passed into the uploader (for uploading) via a new public function.
  • #482 – Specify custom hover & focus classes for upload button. Thanks to obiwanus for this contribution.
  • #380 – Support uploading folders via drag & drop. Please read my blog post on this feature before asking any questions.
  • #113 – Allow parameters to be passed in request body only (via a new option). Please read my blog post on this feature.

Bugs Fixed

  • #511 – Multipart encoded requests include redundant filename parameter, in query string. Thanks to AnilRh for reporting this.
  • #498 – jQuery object parameters passed to functions are not transformed to HTMLElements
  • #493 – After resetting uploader, event handlers are added again, but not removed. This results in redundant event handlers attached to elements in some cases.
  • #492 – jQuery selectors are not obeyed when binding plug-in instance. Only the first matched element is used. Thanks to akre54 for reporting this.
  • #474 – Caught callback error message sometimes lists the wrong callback as the source of the failure.
  • #220 – Dragging an IMG element or text node enables drop area. Thanks to donaldallen for reporting this.
  • untracked – Public cancel function in FineUploader does not remove the associated file element in the UI.

Breaking Changes

The template option has changed. I needed to add some elements to display the status of the uploader when processing large dropped directories or dropped directories behind high-latency connections. Please have a look at the current template option default value and modify your code accordingly if you have contributed a modified template option. UPDATE: Templating changed drastically in Fine Uploader 4.0. If you are using 4.0, please see the styling documentation page for details.

Note About The Codebase

There were also many changes made to the code that are not visible in the features and bugfixes listed above. I am continuing to work on making the Fine Uploader codebase JSLint-compliant. Also, as I do this, I’m also attempting to modify the code so a common code style is used throughout the project. This should make life easier for me and forkers. I’m also slowly building up the qQuery section of the uploader with common and useful functions.

Just a reminder – do not make use of any variables or functions that start with an underscore. Those are private, by convention, and may change or disappear at any time. One of my goals is to eventually eliminate this convention in favor of proper information hiding. In 3.1, I split all of the drag and drop code into a new file, ensured it was JSLint-compliant, and removed all of the underscored variables and functions by making use of the module pattern.

What’s In Store For 3.2?

The big feature in 3.2 will be file chunking, which I aim to make fully configurable. This will certainly complement the auto/manual failed upload retry feature. In other words, if a large file fails somewhere near the end of the file, Fine Uploader can simply retry starting with the last failed chunk, instead of re-sending the entire file again. File chunking will be an advanced feature for advanced developers/users of the library, as it will require a good understanding of the process in order to properly code your server-side handler.

The file chunking feature will also allow me to provide a “resume upload” feature. This will allow your users, if enabled, to resume a paused or failed upload from a previous session. Let’s say a user is uploading a very large file from their laptop at a coffee shop, and, for whatever reason, they have to leave before the upload has completed. This feature will allow them to resume the upload on their laptop when they get home, simply by attempting to upload the file again (via drop or “select files” dialog). I haven’t worked out any of the details yet, so the specifics surrounding the resume may change, but this is the general idea, at this point.

You can follow progress of file chunking and upload resume in the issue tracker.

As always, please let me know (in the forums or the issue tracker) if you have any suggestions for improvement, or any killer features you’d like me to add.

-Ray Nicholus

setParams is now much more useful in 3.1

It seems that the old setParams function caused a lot of confusion among developers utilizing Fine Uploader in their projects. Historically, the setParams function could only be called in your onSubmit callback. Any calls after this point would result in changes to the parameters for the next file sent, and not the current file. Furthermore, calling the setParams function would result in replacement of the params passed along with ALL files after that point. If you only wanted to change the parameters for one file, you would have to call setParams again the NEXT time you handle an onSubmit callback, passing in the original parameters.

3.1 brings along some much needed improvements to the setParams function. Also, my changes will NOT break any existing code. If you, for some reason, like the way setParams has functioned up until now, or you simply don’t want to adjust your code to take advantage of the improvements, you can leave your code alone any everything should work as it did before.

Change parameters at ANY point

Now, you can utilize setParams to change parameters passed at any point in the upload process. For the current file, you can adjust the parameters passed along with the request as late as the onUpload callback. You can also adjust parameters for a file in onManualRetry and onAutoRetry callbacks.

Easily change parameters only for a specific file

The setParams function has been improved to take an optional file ID parameter. So, if you only want to change the parameters passed for a specific file, simply pass your parameters object, along with the file ID to the setParams function. This may be especially useful for users that have the autoUpload parameter set to false. If you want to change parameters for all files, you can still do that by omitting the file ID parameter.
The new signature is setParams(Object params, [optional] Integer fileId).

Notes:

  • Once you change parameters for a specific file by specifying its ID in your setParams call, calls to setParams without a file ID will NOT affect this file. If you want to adjust parameters AGAIN for such a file, you will need to call setParams again, specifying that file’s ID.
  • In 3.1, the file ID’s for all internal uploaders are now consistent. Previously, the form uploader would use a string ID, such as “qq-uploader-handler-iframe” followed by a number. The XHR uploader would simply use a number for IDs. Now, both internal uploaders simply use a number, so you can expect that the IDs for all files will always be integers.

As always, please let me know if you would like any further improvements, or if you have any concerns or questions.

-Ray Nicholus

Include params in the request body OR the query string

Updates since the original post/feature implementation

This is a new feature for 3.1.  I completed it sooner than originally scheduled due to the number of requests I received for this feature.  For backwards compatibility reasons, all parameters will be specified in the query string by default, but you can override this by setting the paramsInBody property (inside the request option).  If this option is set, all parameters will be sent in the request body only.  Note that this will all force all requests to be multipart encoded, so you will need to account for this in your server code appropriately.

Please note that all parameters, whether they are included in the query string or in the request body (no longer applies to request body params as of 3.3), are URI component encoded.  This means that you must decode the parameter keys and values server-side.  Javascript’s encodeURIComponent function, which is used to encode the parameters client-side, uses the UTF-8 encoding scheme.  So, you must ensure the same encoding is used server-side.  Also, be sure to set the charset meta tag in your HTML.

Fine Uploader params are powerful

The request params option of Fine Uploader is actually quite powerful.  It may be useful for you to understand its full potential.

Simple key-values

The most basic and common parameters sent along with each upload request will be simple key-value pairs: the key being a string and the value a number or string.  For example:

var uploader = new qq.FineUploader({
    request: {
        endpoint: 'upload/handler',
        params: {
            one: 1,
            two: 'two',
            'three': 'three'
        }
    }
});

The parameters will be passed either in the query string, or the body of the request (depending on the value of the paramsInBody option).  In this case, 3 parameters will be passed, with keys of “one”, “two”, and “three”, along with values of “1”, “two” and “three” respectively.

Nested objects

You may also pass objects as values.  For example:
var uploader = new qq.FineUploader({
    request: {
        endpoint: 'upload/handler',
        params: {
            foo: {
                one: 'bar',
                two: 'bar2'
            }
        }
    }
});
Fine Uploader ensures the context of each parameter is kept intact.  So, sever-side, you will see a key of “foo[one]” with a value of “bar” and a key of “foo[two]” with a value of “bar2”.  Fine Uploader supports multiple levels of object nesting.

Functions as values

Finally, you may include functions as parameter values.  Take the following:
var fileNum = 0,
    uploader = new qq.FineUploader({
        request: {
            endpoint: 'upload/handler',
            params: {
                fileNum: function() {
                    return fileNum++;
                }
            }
        }
    });
Note that this function will be evaluated for each file that is submitted.  So, along with the first file request, a parameter of “fileNum” will include a value of “0”.  The next file will have the same param, this time with a value of “1”.  Please understand that any params with a function value must return either a number or a string, and not an object.
As always, if you have any requests to improve this feature, or any feedback at all, feel free to let me know.
-Ray Nicholus