UPDATE: I found 2 bugs in upload progress module. If you have already installed. update to at least 0.1 version: http://github.com/drogus/apache-upload-progress-module/commits/0.1
I’ve installed mod passenger on my server recently. It’s really great software. Now I don’t have to worry about monitoring, nginx proxy, load balancing, big file uploads… and it’s fast! With Ruby Enterprise Edition it’s even faster.
Personally I don’t care about people saying that phusion wants to promote themselves on REE as long as it gives faster ruby with lower memory use (but yes, I know, REE is not best choice for a name :).
After installing I’ve realised that my shiny upload progress bar (thanks to Upload Progress Module for nginx) was not working. Oh gods! What a tragedy!
But fear not. I’ve written apache upload progress module to have my lovely progress bar back again. As a lazy developer I’ve implemented reports in the same way as in nginx upload progress, so my applications are working without changing any signle line of code. If you were using nginx upload progress just drop the module, change your config and you’re good to go :)
I’m testing it in one of my production servers, but be carefull – it’s not well tested in other enviroments (i have gentoo with apache 2.2.8). Any feedback will be helpfull. Give me a note in comments if you encounter any problems.
So you want to be cool and have your own sexy progress bar in your app? Keep reading ;)
To install module you must download it using git:git clone git://github.com/drogus/apache-upload-progress-module.gitapxs2 -c -i -a mod_upload_progress.c- -c is for compiling
- -i is for installing (copy mod_upload_progress.so to apache library dir)
- -a is for activating (add LoadModule option into your apache conf file)
LoadModule upload_progress_module path/to/apache-upload-progress-module/.libs/mod_upload_progress.soUploadProgressSharedMemorySize 1024000This sets shared memory size to 1M. By default it’s 100kB.
To add tracking and reporting upload for a virtual host in apache you will need to add:
<Location />
# enable tracking uploads in /
TrackUploads On
</Location>
<Location /progress>
# enable upload progress reports in /progress
ReportUploads On
</Location>
Now all uploads will be tracked and reports are under /progress
Format of the report is JSON. From nginx wiki:
The returned document is a JSON text with the possible 4 results:
- the upload request hasn’t been registered yet or is unknown:
new Object({ ‘state’ : ‘starting’ })
- the upload request has ended:
new Object({ ‘state’ : ‘done’ })
- the upload request generated an HTTP error:
new Object({ ‘state’ : ‘error’, ‘status’ :
}) One error code that is interesting to track for clients is HTTP error 413 (Request entity too large)
- the upload request is in progress:
new Object({ ‘state’ : ‘uploading’, ‘received’ :
, ‘size’ : }) The HTTP request to this location must have either an X-Progress-ID parameter or X-Progress-ID HTTP header containing the unique identifier as specified in your upload/POST request to the relevant tracked zone. If you are using the X-Progress-ID as a query-string parameter, ensure it is the LAST argument in the URL.
Now the last thing to do is to implement progress bar. I don’t like repeating others and there is great tutorial on setting up upload progress bar with nginx and merb
UPDATE: I released jquery upload progress library with Safari 3 support. More info here. UPDATE2: I’ve upgraded prototype version to work in Safari.
It’s for merb and nginx but if you drop the scripts in your rails app and with apache-upload-progress-module it will work. :) Basically if you have your own code handling uploads (for example using attachment_fu) you can just add javascript and css – it’s unobtrusive.
If you’re using prototype I’ve rewritten script and made a demo. You can also grab files
I hope you enjoy this article. Progress bar is in my opinion one of the most useful technics – there is nothing more annoying than large file uploading without any info on state of an upload.
pimpmaster said
Jun 25, 2008 @ 09:23 PM
Nice writeup! :)
Your demo is busted in Safari 3 though (No progress is shown) :(
Micha said
Jun 26, 2008 @ 10:41 AM
For some reason, I get an error in my error_log: [error] (12)Cannot allocate memory: Upload Progress cache: could not create shared memory segment Configuration Failed
This error can be avoided by setting UploadProgressSharedMemorySize to something less than ~75000.
My Apache is Server version: Apache/2.2.8 (Unix) Server built: Mar 4 2008 21:37:02 on MacOS 10.5 (the builtin one) and was compiled with http://pastie.org/222492
Any thoughts?
Drogomir said
Jun 26, 2008 @ 11:16 AM
pimpmaster: I will check it :)
Micha: I don’t know what causes this problem but I changed default shared memory size to 50kB – it will be safer then 100kB. You can change shared memory size even to 10240 (10kB) if you don’t provide hundreds of uploads simultaneously it’s not a problem – it’s only for keeping upload data (key, upload size and size of received bytes). Upload data is removed from memory 60s after upload has ended.
Ziggy said
Jun 26, 2008 @ 09:02 PM
Does this work with https ? I can’t seem to get it working on FC8/x86_64, the /progress keeps reporting ‘starting’ .... httpd-2.2.8-1.fc8/php-5.2.4-3
Ziggy said
Jun 26, 2008 @ 09:59 PM
Doesn’t seem to matter if it’s https or not, the /progress is stuck on ‘starting’ anyways (FC8 etc.)
Drogomir said
Jun 27, 2008 @ 12:16 AM
Ziggy: Did you provide existing X-Progress-ID? It is in “starting” state if it can’t find valid upload data with given X-Progress-ID.
Ted said
Jun 27, 2008 @ 06:52 PM
Confirm not working in safari 3. Seems like a JS issue? =/
Drogomir said
Jun 28, 2008 @ 10:34 PM
I’ve checked it in safari on windows and it seems to be problem with sending ajax request while page is loading. I will investigate this one :)
Michele said
Jun 29, 2008 @ 01:02 AM
Yep, it definitely seems to be a problem with sending an AJAX request while loading a page (indeed, if you manually enter the AJAX call in the address bar prepending javascript: the call is executed correctly, but then it stops sending the main request).
I imagine it should be possible to fix this by using an iframe to separate the two requests…I’ll play with it tomorrow.
Meanwhile, thanks for the module: it has really helped! :)
Michele said
Jun 29, 2008 @ 12:36 PM
I managed to make it work in Safari, too: you need to insert an iframe in the page, give it a name (e.g. name=”progress”) and have it load a static HTML page which you’ve carefully crafted with: basic HTML structure (making sure you load all JS libraries you need) and in the body you define a function with the AJAX call and logic to update the progress bar. Of course, this function needs to passed the uuid.
Then from the main page you call name_given_to_the_iframe.name_given_to_the_function(uuid) In the function you can access the main page DOM object like so: parent.document.getElementById()
Drogomir said
Jun 29, 2008 @ 04:26 PM
Michele:
Thanks for checking it. I will update my example shortly :)
I will add Safari hacks to upload-progress script: http://github.com/drogus/jquery-upload-progress/tree/master and possibly make prototype version with some tweaks like in jQuery version.
Michele said
Jun 29, 2008 @ 04:57 PM
Just let me know if there’s anything not clear in my explanation. :)
Drogomir said
Jun 30, 2008 @ 03:03 AM
I’ve added safari support to jquery version of upload progress script: http://github.com/drogus/jquery-upload-progress/tree/master . Tomorrow I will try to code prototype version with callbacks and other useful stuff.
Check new demo which is supposed to work in safari: http://drogomir.com/files/blog/jquery-upload-progress/example
Matze said
Aug 20, 2008 @ 01:19 AM
Hi, i just installed upload_progress_bar and my Progressbar stopps counting after about 250000Bytes received. Upload still works and the status changed to done after finishing upload.
My Setup: Apache 2.2.9, mod_rails (Passenger) 2.0.3 and apache-mod-upload-progress (installed yesterday)
Any Ideas?
btw. thanks for the apache-mod!
Drogomir said
Aug 20, 2008 @ 07:20 AM
Matze:
Which browser did you use? While using Firefox 3 I have similar problems. Try to clear browser cache when the file is uploading and see if progress bar is still stuck. If yes, then it’s browser problem. If no, could you enable debug messages in apache and see if anything weird is reported in the log file?
Matze said
Aug 20, 2008 @ 01:13 PM
hmmm… indeed, it’s working on my Windows-Box with IE 7, FF2 and FF3. On my OS X (10.5.4) with FF3 and FF2 i had the above mentioned problem.
Sorry for the quick post without testing on an other box and thanks for your quick reply!
Maybe it’s a problem with caching, i will test it coming weekend.
Matze
OWL said
Aug 25, 2008 @ 01:51 AM
Hi, Thanks for module. Works fine with IE and FF but does not work with Opera. checked with opera = 9.51 under winxp.
Looks like Opera can’t do Ajax request during file upload. Does anybody know a hack for Opera ?
OWL said
Aug 25, 2008 @ 01:51 AM
Hi, Thanks for module. Works fine with IE and FF but does not work with Opera. checked with opera = 9.51 under winxp.
Looks like Opera can’t do Ajax request during file upload. Does anybody know a hack for Opera ?
Drogomir said
Aug 25, 2008 @ 10:50 AM
It works in Opera 9.5 for me (Ubuntu). I will check 9.51
If it can’t send ajax request it is the same issue that safari has.
Adam said
Aug 28, 2008 @ 04:33 AM
Hi There,Thanks for this patch. its awesome. Can anyone explain whether using this combined with mod_rails will get past the blocking issue with rails uploads? ie, will apache handle the upload and then pass it to rails or is rails handling the whole thing ??
Drogomir said
Aug 30, 2008 @ 01:17 PM
About the Opera: I’ve checked 9.52 and it is not working for me either. Hacks for safari are not working, so I will have to debug it on Opera.
Adam: mod_passenger is handling uploads – it’s faster and it doesn’t block rails instance. If you want to do something while uploading you must use Apache filters (like in my upload progress module)
Michael said
Sep 19, 2008 @ 07:12 PM
This is a great patch, but I’ve run into a problem. I had it installed, working beautifully (like I said, it’s great), very happy, etc, etc—then I installed an SSL cert. The httpd.conf directives in my Virtual Host blocks no longer were obeyed when viewing through HTTPS. I can move the directives outside a virtual host, but then your plugin becomes buggy: ’/progress’ works fine before you start an upload (state:starting), but when you try to upload a file, the upload script runs forever, as does the ’/progress’ feed.
I’m kind of clueless regarding SSL, so perhaps I’m missing something. Any suggestions?
Drogomir said
Sep 21, 2008 @ 02:49 PM
Michael: I will have to test plugin and scripts with ssl. However I don’t have much time right now, so I’m afraid that it can last some time. Could you pastie logs with SSL turned on and possibly log from safari error console?
KTU said
Sep 26, 2008 @ 08:39 PM
This looks very promising one but I haven’t been able to make it work.
I was looking progress bar with RAILS form page which could pop-up window for the progress bar and show percentage how upload is progressing. Is this possible with the current tool ?
Any change to get full working demo from somewhere ? Yes I downloaded examples but it wont work is complaining about various javascript errors etc…
Thank You
Drogomir said
Sep 27, 2008 @ 03:05 AM
Here is example in jquery: http://drogomir.com/files/blog/jquery-upload-progress/example/
Could you paste those errors?
mikehal said
Sep 28, 2008 @ 09:17 AM
I’m considering your module – appreciate some guidance before I jump in:
my setup: OS X.5 Apache 2 and php 5.2.current. No prototype or jquery or yahoo stuff. I would like to be able to upload the file from an iframe, pass the error data and image path back to parent.js via an object literal and have the js call for the upload progress data on a setinterval httprequest and modify the width of a div in the iframe accordingly.
what am I calling exacty? will this module work for me? do I need the passenger/rails stuff? what is apxs2? apxs compiles it OK so long as I install and configure manually it looks like it might work. not sure how to test it though.
thanks!
Drogomir said
Sep 28, 2008 @ 01:45 PM
mikehal:
If you don’t use jquery/prototype you can still use upload progress, but you will have to write your own code that will get progress from server (it is served as JSON). If you’re not sure how it should look like, you can check jquery/prototype upload progress plugin sources or even some js code written to show lighttpd upload progress (apache upload progress has the same response format so it will work :).
Scenerio that you have desribed seems to look ok, so this module will work for you :)
You don’t need rails/passenger stuff. It only requires apache :)
apxs2 is tool to compile modules for Apache2. As long as it compiles and works you can use apxs or even gcc (but you will need to pass some options then).
The easier way to test it is to set it for one of virtual hosts and run one of examples from jquery/prototype upload progress module :) They are in the packages. But testing it locally is a little bit hard, cause files are uploading so fast that you won’t notice progress, so I recommend doing it on some remote server. Check post about javascript plugins – there are links for examples and packages (examples are in packages :)
mikehal said
Sep 30, 2008 @ 04:19 AM
Drogomir:
Thanks for the guidance. I have it compiled and I manually put the .so in the libexec dir and added it to httpd.conf. I was running Apache2 out-of-the-box so at first I just tried:
http://localhost/progress?X-Progress-ID=123
but got a 404 until I added the vhost stuff. Then I got:
new Object({ ‘state’ : ‘starting’ })
which got me excited. After hooking it up (an upload form with a hidden element for the UID; httpRequest/progress loop via an onSubmit) I cannot get anything more back – always state:starting. testing both locally and remote.
the prototype-assisted demo is not doing any better. stumped for now. What am I missing?
will recompile with gcc etc.
Drogomir said
Sep 30, 2008 @ 02:14 PM
mikehal:
Could you pastie apache error log (with log leve debug) ? Which browser have you used for this tests? Try jquery or prototype demo on firefox with firebug and check requests in console.
mikehal said
Sep 30, 2008 @ 06:14 PM
Drogomir:
Same result FF3 and SAF3. for bothe remote and local access to the same files. your prototype demo is working remotely , just not locally. Here is the remote access debug log for my stuff followed by your prototype-assisted demo remote access followed by yours hit locally (it hangs)
[debug] mod_upload_progress.c(139): Upload Progress: Upload in trackable location: /processors/upload.php. [debug] mod_upload_progress.c(682): Upload Progress: Found id=f264a380e4e901e10b5e2399f209a472 in location with reports enables. uri=/progress [debug] mod_upload_progress.c(707): Node with id=f264a380e4e901e10b5e2399f209a472 not found for report
yours via remote
[debug] mod_upload_progress.c(139): Upload Progress: Upload in trackable location: /drogus/example/index.html. [debug] mod_upload_progress.c(143): Upload Progress: Progress id found: 494b24386a970469e70d4c70d418c6c4. [debug] mod_upload_progress.c(369): Upload Progress: Inserted node into an empty list. [debug] mod_upload_progress.c(462): Upload Progress: Added upload with id=494b24386a970469e70d4c70d418c6c4 to list. [debug] mod_upload_progress.c(682): Upload Progress: Found id=494b24386a970469e70d4c70d418c6c4 in location with reports enables. uri=/progress [debug] mod_upload_progress.c(697): Node with id=494b24386a970469e70d4c70d418c6c4 found for report
yours locally (same server, upload file etc)
[debug] mod_upload_progress.c(139): Upload Progress: Upload in trackable location: /drogus/example/index.html. [debug] mod_upload_progress.c(143): Upload Progress: Progress id found: fd8bdd71016085e791c0056e759e797f. [debug] mod_upload_progress.c(374): Upload Progress: Inserted node at the end of the list. [debug] mod_upload_progress.c(462): Upload Progress: Added upload with id=fd8bdd71016085e791c0056e759e797f to list.
Obviously if your prototype is working the module compiled OK and is loaded. I’m choking it with my script somehow.
mikehal said
Oct 01, 2008 @ 01:13 AM
Looks like assigning the uid via a hidden form field is a problem. (the PECL uploadprogress PHP ext assigns it this way).
Drogomir said
Oct 01, 2008 @ 12:54 PM
Unfortunately none of those upload progress modules (for nginx, lighttpd and mine for apache) gets UID from form fields. Currently you can only attach it to address as GET parameter or send in header X-Progress-ID. If PECL assigns it this way, I will consider adding possibility to POST parameters, but as I said before I don’t have much time, so I think that it won’t be quick…
Rick said
Oct 02, 2008 @ 02:31 AM
I successfully compiled this on OS X.5.5. However, apachectl does not like the resulting shared lib.
httpd: Syntax error on line 118 of /private/etc/apache2/httpd.conf: Cannot load /usr/libexec/apache2/mod_upload_progress.so into server: dlopen(/usr/libexec/apache2/mod_upload_progress.so, 10): no suitable image found. Did find:\n\t/usr/libexec/apache2/mod_upload_progress.so: mach-o, but wrong architecture
I am using the default Apache 2.2 on OS X with its apxs command. Any ideas?
mikehal said
Oct 02, 2008 @ 09:26 PM
Rick: I ran into the same problem with the same setup so I flagged apxs to compile only and moved the resulting .so from the .libs dir to /usr/libexec/apache2 by hand and added the load module directive to httpd.conf. It’s working! (don’t forget to add in the vhost directive per Drogomir’s guide).
Rick said
Oct 02, 2008 @ 10:08 PM
mikehal: Thanks for the suggestion. I did the same on my box and I am still getting the error. I will have to try this on one of my Linux boxes. OS X has not been working too well for me since the upgrade to 10.5.5.
mikehal said
Oct 03, 2008 @ 10:30 PM
Drogomir:
I’ve had some time to test out what’s coming back from /Progress. A couple of questions/observations:
1. It looks like all other kv’s are wiped when state hits ‘done’. ?
If the polling interval is too short – say sub 300ms then the received kv comes back irregularly – one or two reports early on a 1.5MB file then nothing until complete. At 500 it’s smooth. But small files upload before the second poll occurs and I want to get at size: or received even if I’m late but the properties are undefined.
2. Can Apache be directed [from a module] to call a specific js function? Just curious. Not exactly sure what I would do with it. (There’s a slight delay before /progress becomes available)
Drogomir said
Oct 04, 2008 @ 02:51 PM
What do you mean by kv?
Irrgularity of responses with short interval is caused by uploading file. Normally, response from server should be very quick (it’s small and generated in a few miliseconds), but while uploading, responses are much slower, so you have to wait longer. If you want to provide progress bar for smaller files you can consider swfupload – it looks great out of the box and you don’t have to fight with all those ajax issues. Ofcourse there are also bad sides: it is stuck at 0% on linux (and it’s very slow for me), it’s much more obtrusive and there is more people with disabled flash than people without javascript.
If it fits your needs you can give it a try.
2. Apache is server side and javascript is client side so basically it can’t call any javascript (there are server side js projects like jaxer but for keeping things simple let’s forget about it for now). I think that your question should be – Can apache force javascript to call a specific function?
The answer is: yes, it can. If apache will response with type ‘text/javascript’ it will be executed. But technically it’s pretty much the same as calling a function in js after getting JSON response.
The other way to do it (and it’s probably closer to what you described) is a push server. Rather than calling xhr requests with given interval server forces javascript to make a request.
mikehal said
Oct 05, 2008 @ 02:12 AM
By kv’s I meant the key-value pairs coming in from /progress. I guess I should say properties? When state:’done’ , size: and received: are no longer defined. I have to store them ahead of state:done which for a really fast upload is touchy. I’m new at this stuff so forgive me if this is idiotic. ;)
mikehal said
Oct 05, 2008 @ 02:29 AM
per swfupload: I dug into that and got it working-ish but it, well, led me to your module if you know what I mean. Tricked out OOP js libraries tied into css definitions hooked up to some actionscript complied into an invisible ‘movie’ hurt my head.
George Moschovitis said
Oct 05, 2008 @ 03:21 AM
Does this module actually save the upload to a tempfile?
Drogomir said
Oct 05, 2008 @ 04:40 PM
mikehal: I think it’s not idiotic, but I don’t understand your question :) Could you give some examples or explain again?
George Moschovitis: this module is something like filter working while data is uploading. It only gives you information about upload. After data is uploaded you have to handle it yourself.
Tim said
Oct 08, 2008 @ 06:56 PM
Rick, mikehal -
It seems that apsx on the mac only compiles for one architecture… i.e. not a universal binary. I got around the load error mentioned by explicitly compiling all architectures like so:
sudo apxs -c -i -Wc,-arch -Wc,ppc7400 -Wl,-arch -Wl,ppc7400 -Wc,-arch -Wc,ppc64 -Wl,-arch -Wl,ppc64 -Wc,-arch -Wc,x86_64 -Wl,-arch -Wl,x86_64 -Wc,-arch -Wc,i386 -Wl,-arch -Wl,i386 mod_upload_progress.c
Maybe this should be mentioned in the readme?
(... and, yeah, that is one fugly command line grin)
HTH, Tim
Tim said
Oct 10, 2008 @ 04:52 AM
Out of curiosity, how long does upload info persist after the upload completes?
Drogomir said
Oct 10, 2008 @ 04:25 PM
Tim:
I will add your findings to README file.
Upload info expires after 60 seconds, but is cleaned after another upload starts (I’ve done it in the simple way – nodes are checked for expiry only when new node has to be tracked).
Attila said
Oct 30, 2008 @ 02:51 AM
Tim,
you could still add the -a parameter as well, I did it that way and I have no problems with this so far on OS X. Thank you for sharing your solution, apparently it helped a lot. Also, thanks to Drogomir for this great software. I’ll test the hell out of it now :)
Tim said
Nov 10, 2008 @ 04:23 PM
Attila -
I did try with the -a flag and for some reason it only compiled the i386 part and for some reason (I am not entirely sure why…) Leopard’s apache expects the x86_64 code to be there as well.
Maybe a bug in apxs?
Cheers, Tim
odorryagrinty said
Nov 14, 2008 @ 07:28 AM
Hello! Good site, much useful
Yves said
Nov 24, 2008 @ 08:48 PM
FYI – it does not work with Debian Etch and Apache 2.2 MPM Worker (Threaded)
Forcing reload of web server (apache2)... waiting ..apache2: Syntax error on line 185 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/upload_progress.load: Can’t locate API module structure `upload_progress’ in file /usr/lib/apache2/modules/mod_upload_progress.so: /usr/lib/apache2/modules/mod_upload_progress.so: undefined symbol: upload_progress failed!
Yves said
Nov 24, 2008 @ 08:55 PM
Arghs… it works. Just do it this way on Debian.
Yves said
Nov 24, 2008 @ 11:41 PM
Just to get it right… the X-Progress-ID is generated randomly and prevents users from polling foreign upload progress?
Drogomir said
Nov 25, 2008 @ 02:52 AM
Yves: Yes, you’re right – you need X-Progress-ID to identify current upload.
knodi said
Nov 30, 2008 @ 01:54 PM
I have a problem that the done state is never received it goes straight from uploading to error even thou the upload is done properly the done state is never sent.
is there something I need to do to get the done state?
how does this work with multiple downloads at the same time/cluster environment?
Drogomir said
Dec 01, 2008 @ 07:10 PM
knodi: could you set log level to debug and pastie some of debug messages?
Done state should be set just after upload is complete, you don’t have to do anything.
What do you mean by “multiple downloads”?
knodi said
Dec 02, 2008 @ 10:51 AM
@Drogomir
I got the done state now that added the UploadProgressSharedMemorySize 1024000.
But there is a more sinister problem that I came across “* Unexpected error in Passenger: The HTTP client sent incomplete upload data.” error.
found more info on it here http://groups.google.com/group/phusion-passenger/browse_thread/thread/4f496e3fde297a48
don’t know if its passenger or UploadProgress. But it happens only with both are used together.
Knodi said
Dec 02, 2008 @ 10:53 AM
@Drogomir
by multiple downloads I mean in a apache clustered environment.
Knodi said
Dec 04, 2008 @ 06:37 PM
Ignore my post about ““* Unexpected error in Passenger: The HTTP client sent incomplete upload data.” This seems to be a problem with Passenger (mod_rails) only on OSX.
One more thing how does this work in a apache cluster environment. Work my front(load balancer) apache server be able to get the /progress or does the apache that the load balancer apache proxies the request the only apache process able to reply with the correct /progress info.
To visualize this: lets say i have 1 load balancer apache(Lets call it Apache A) that proxies request to 2 other apache processes lets call them Apache B and Apache C.
So when the upload file post comes in Apache A will hand the request of the Apache B after this happens when I do /progress Apache A hands the request to Apache C as Apache B is busy with the upload request. So this will cause an error as Apache C doesn’t know anything about the upload that going on.
So my question is this an fair interpretation of how upload progress mod will work?
Drogomir said
Dec 06, 2008 @ 02:33 PM
Knodi: Yes, unfortunately you’re right. Upload data is shared only between vhosts. In case of such configuration the easiest way is to use some client side upload progress (SWFUpload maybe).
RSS feed for comments on this post
Leave a Comment