Corrupt files, when stored in the DB, often symptomatic of encoding issues with your database. Is this a new 6.3 installation, or did you upgrade from an earlier version. See http://doc.tiki.org/Upgrade#Fix_the_encoding_issue for details.
Since you're storing files in the database, you may also need to increase the max_execution_time and max_input_time.
Other things to check:
- Are you using wiki attachments? If so, are you storing them in the database, too? And if so, do you also encounter issues with attachments?
- If you temporarily change to store files in the file directory, are you able to upload/download large files? If so, then the problem is most likely that MySQL is closing the database connection before Tiki completes the file upload.