Baidu - How to fix "insufficient data written" errors

tzverifier

Hey there, I'm trying to get some files out of my Pan Baidu account (to google drive). The files are on the order of 500MB to 1.5GB. For each file, the transfer runs for several hours, and then fails, with "insufficient data written" as the only error. I've found this other thread about this, but it's from 2016: https://forum.multcloud.com/discussion/324/moving-files-out-of-baidu-a-nightmare In it, the poster says it eventually works after several tries, so I guess I can keep retrying, but I'm hoping there's been some solutions in the past 4 years. Is this just caused by the Great Firewall, meaning even just using baidu netdisk would be this slow and unreliable?

Admin

As for "insufficient data written" issue,  it seems that Google official website itself has this problem so we can not resolve it as the API is provided by Google. So please try again at another time.

tzverifier

Google is working fine, and works great with any other transfer source. I bet Baidu is working fine as well, and just throttles throughput down, likely in high-latency bursts. If I had to guess, I'd guess that what's happening is:

1. MultCloud opens a read buffer from baidu and a write buffer to google, and just passes bytes along.
2. The google buffer rightfully has a timeout if left open with no data coming in.
3. Because data from baidu can be scarce, it sometimes triggers the google timeout while it waits for more data.

Basically, this is like "buffer underrun" errors from old school CD writers.

The solution would be to have a larger buffer between your source (Baidu) and your sink (Google), and for your software to be aware of how it's performing, and _slowly_ drain the buffer into google, to minimize instances of these buffer underruns. There's a chance you're already doing this. That could be why it takes several minutes to start to see any progress. In that case, it could be just a matter of adjusting your tuning parameters to allow for longer baidu slowdowns.

ALSO:  wonder if there are other cloud providers in China that are easy to set up accounts on that you support, and that don't throttle like Baidu does. It could then be more reliable for me to go:

Baidu --> (other china cloud drive) --> Google Drive

That way, the hop across the GFW is through a more forgiving provider. What do you recommend?

Admin

As for the "insufficient data written" issue, we can't confirm the exact reason that caused the problem. But, it indeed is related to the API of google drive. Could you run the transfer or sync task again?
And, sorry that we currently don't support other clouds of providers in China.

tzverifier

No worries!

I retried a number of times, and things eventually worked out. It seems to be more reliable during times that correspond to night time in China.