gcloud writestreams no longer working when piping stdout from GraphicsMagick

[jseeley78]

GraphicsMagick lib:
http://aheckmann.github.io/gm/

ex:

var gm = require('gm').subClass({ imageMagick: true }),
      src = '/src/file.jpg',
      dst = myBucket.file('file.png');

var wr = dst.createWriteStream();

gm(src)
    .setFormat('png')
    .stream(function(err, stdout, stderr) {
    if (err) return cb(err);
    // pipe output to writestream
    stdout.pipe(wr);
    wr.on("error", function(err) {
        return cb(err);
    });
     wr.on("finish", function() {
        return cb();
    });
});

works in release 20 and older(using complete event); breaks in release 21, finish doesn't seem to trigger, error is fired with->
{ [Error: The uploaded data did not match the data from the server. As a precaution, the file has been deleted. To be sure the content is the same, you should try uploading the file again.] code: 'FILE_NO_UPLOAD', errors: [ null ] }

[stephenplusplus]

Thanks! I'll look into this right away.

[stephenplusplus]

I was able to use the same code and get the file to upload successfully. finish won't emit because error does, and it's an error because somehow, the hash of the bytes that were sent upstream aren't matching what the API says it received.

To get around this, you can turn off validation, or limit it to a specific type. Playing with these options might help debugging where the problem is happening.

To do that:

// Turn off all data integrity checks
var wr = dst.createWriteStream({ validation: false });

// Use just md5
var wr = dst.createWriteStream({ validation: 'md5' });

// Use just crc32c
var wr = dst.createWriteStream({ validation: 'crc32c' });

If you try this, please let me know which worked (if any), and confirm that the file that ended up in the bucket is properly formed.

[jseeley78]

this seems odd but->

validation: false (fails)
validation: true (succeeds)
validation: 'md5' (succeeds)
validation: 'crc32c' (succeeds)

[stephenplusplus]

I think I caught the bug, though I'm not sure why it failed succeeded when you explicitly provided validation: true, as that's what is defaulted.

Can you try out my patch branch to see if it works:

$ npm install --save stephenplusplus/gcloud-node#spp--storage-stream-fix

[jseeley78]

I can see why it would have failed when validation is set to false, but like you said the validation default is true when you don't specify... (and why would the stdout stream from GraphicsMagick make any difference?) ANYWAY... this fix works ;-)

[stephenplusplus]

Yeah, weird. Well, PR coming soon after I write some tests. Thanks for letting us know about this issue!