My name
is
Jon Skeet

Uploading stream with MD5 hash results in "The Content MD5 you specified was invalid"

I am working on an implementation, using Amazon S3. I use the Amazon C# SDK, and I try to upload a created ZIP file using the putObject method.

When I upload the file, I get the following error:

{Amazon.S3.AmazonS3Exception: The Content-MD5 you specified was invalid

I generate a fine and working memorystream, and I can upload it to Amazon S3 without errors. However, when I provide the following line it gives me problems:

request.MD5Digest = md5;

Am I proving the MD5 the correct way? Is my MD5 generation correct? Or is there something else wrong?

The requirement to my upload code:

Once the Zip file has been created and you have calculated an MD5 sum value of that file, you should 
transfer the file to the AWS S3 bucket identified in the S3Access XML. 
Transfer the file using the AmazonS3, PutObjectRequest and TransferManagerclasses. 
Ensure the following meta data attributes are included via adding an ObjectMetaDataclass instance to the
PutObjectRequest:
• MD5Sum (via setContentMD5)
• Mime ContentType (setContentType)

My upload code

The client.PutObject() gives the error:

public void UploadFile(string bucketName, Stream uploadFileStream, string remoteFileName, string md5)
        {
            using (client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKeyID, secretAccessKeyID, config))
            {
                try
                {
                    StringBuilder stringResp = new StringBuilder();

                    PutObjectRequest request = new PutObjectRequest();
                   // request.MD5Digest = md5;
                    request.BucketName = bucketName;
                    request.InputStream = uploadFileStream;
                    request.Key = remoteFileName;
                    request.MD5Digest = md5;

                    using (S3Response response = client.PutObject(request))
                    {
                        WebHeaderCollection headers = response.Headers;
                        foreach (string key in headers.Keys)
                        {
                            stringResp.AppendLine(string.Format("Key: {0}, value: {1}", key,headers.Get(key).ToString()));
                            //log headers ("Response Header: {0}, Value: {1}", key, headers.Get(key));
                        }
                    }
                }
                catch (AmazonS3Exception amazonS3Exception)
                {
                    if (amazonS3Exception.ErrorCode != null && (amazonS3Exception.ErrorCode.Equals("InvalidAccessKeyId") || amazonS3Exception.ErrorCode.Equals("InvalidSecurity")))
                    {
                        //log exception - ("Please check the provided AWS Credentials.");
                    }
                    else
                    {
                        //log exception -("An error occurred with the message '{0}' when writing an object", amazonS3Exception.Message);
                    }
                }
            }
        }

Overall method

My process method (to see the overall flow). Sorry about the current state of the code, which is more pseudo code than production code:

 public void Process(List<Order> order)
    {
        var zipName = UserName + "-" + DateTime.Now.ToString("yy-MM-dd-hhmmss") + ".zip";
        var zipPath = HttpContext.Current.Server.MapPath("~/Content/zip-fulfillment/" + zipName);

        CreateZip(order, zipPath);


        var s3 = GetS3Access();

        var amazonService = new AmazonS3Service(s3.keyid, s3.secretkey, "s3.amazonaws.com");
        var fileStream = new MemoryStream(HelperMethods.GetBytes(zipPath));
        var md5val = HelperMethods.GetMD5HashFromStream(fileStream);
        fileStream.Position = 0;
        amazonService.UploadFile(s3.bucket, fileStream, zipName, md5val);

        var sqsDoc = DeliveryXml(md5val, s3.bucket, "Test job");

        amazonService.SendSQSMessage(sqsDoc.ToString(), s3.postqueue);
    }

MD5 Hashing method:

This method is used to create a MD5 hash from my memorystream:

    public static string GetMD5HashFromStream(Stream stream)
    {

        MD5 md5 = new MD5CryptoServiceProvider();
        byte[] retVal = md5.ComputeHash(stream);

        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < retVal.Length; i++)
        {
            sb.Append(retVal[i].ToString("x2"));
        }
        return sb.ToString();
    }

EDIT:

Just added fileStream.Position = 0 in the overall method. Still exactly same issue though.

I suspect the problem may be that after computing the hash of the stream, the stream is left at the end of the data... so when anything else reads from it later, there'll be no data. Try add this after the call to GetMD5HashFromStream

fileStream.Position = 0;

This will "rewind" the stream so you can read from it again.

EDIT: Just looked at the documentation, and although the above is a problem, it's not the only one. Currently, you're appending a hex representation of the MD5 hash - but the documentation states:

Content-MD5: The base64-encoded 128-bit MD5 digest of the message

Note the "base64-encoded" part. So you'll want to change your MD5 code to:

public static string GetMD5HashFromStream(Stream stream)
{
    using (MD5 md5 = MD5.Create())
    {
        byte[] hash = md5.ComputeHash(stream);
        return Convert.ToBase64String(hash);
    }
}

See more on this question at Stackoverflow