Excessive 503s from Web Uploader

Troubleshooting
,
1

Hi,

Today We’ve been hitting a lot of 503s and a few 500s when uploading to odrive.

The backend of the upload is Google Cloud Storage.

The 503s and 500s appear randomly.

Below is an 1 request that resulted in a 503

Example

Request Headers

  • Host : proxygen.odrive.com
  • User-Agent : Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:48.0) Gecko/20100101 Firefox/48.0
  • Accept : application/json
  • Accept-Language : en-US,en;q=0.5
  • Accept-Encoding : gzip, deflate, br
  • Cache-Control : no-cache
  • X-Requested-With : XMLHttpRequest
  • Referer : https:// w ww.odrive.com/browse/folder1/06-14-17/folder2
  • Content-Length : 1575265
  • Content-Type : multipart/form-data; boundary=---------------------------575288375900339831841762855
  • Origin : https:// w ww.odrive.com
  • Connection : keep-alive

Note Referer and Origin have extra spaces in there, otherwise the ticket would not let me post.

Path: /uploading?options=&spaceOid=&snodeId=&spaceUserLink=&rootSnodeId=

Response

  • Status: 503

Timing

DNS Lookup: 0ms
Connecting: 12ms
Sending: 0ms
Waiting: 1.77m
Receiving: 0ms

Misc

Firefox is resolving proxygen.odrive.com to : 50.7.108.98

Other Failures

  • Range from 30s to 1.48min request time
  • 14 503s, 5 500s
  • out of 35 requests total.

Google Cloud Storage (https://status.cloud.google.com/) is not reporting any errors. I have saved the HAR if you need more details.

Why is proxygen 503ing and 500ing randomly on file uploads? What can be done to prevent it.

2

Hi @keith,
I had the ops team take a look, but there aren’t any obvious issues being seen on the backend.

Are you dragging files into the uploader via the odrive web client?
How many files are you uploading at a time, and how many files, total?
Are you seeing any issues when uploading via the odrive desktop client?
Is this in an odrive Space?

3

Files are being dragged onto the webclient.

There are 15-50 files in one “drop”. They range from 1-3MB in size each. We did refresh the odrive page before trying.

We have not yet tried the odrive desktop client. Did try dropbox, which worked fine.

It is indeed a shared odrive space that is being uploaded to. (Its not my account that is uploading by who I’m sharing the space with).

The UI shows the upload progress reach ~100%, then it hangs. Then finally fails.

I have the HAR from all the requests if you need more data to track down failed requests.

4

Hi @keith,
There’s a possibility that the uploader is being rejected for too many concurrent connections. Do you know what the external IP is of the box that is doing the uploading?

I just tested uploading 500 4MB files via the web interface to Google Cloud and didn’t get any errors.

5

Is there a way to PM you it?

6

Click on my name and select “Message”

7

Maybe my forum account is too young? I see your post count, but not a message button…

If it helps, here are the first 3 segments of the ip… 162.255.158.XXX

8

Ahh, yeah the message option has a bit of a gate and isn’t available to brand new users.

In any case that IP prefix was sufficient. Are you still seeing the issue. It looks like recent activity from that IP is getting all 200s for the past couple of hours.

9

Last few hours its been better, i’ve been told…
However, I would like to know how what happened to cause those 500 and 503.
If it was rate limiting a 429 would be more appropriate… Does the desktop client not rate limit?
Also, are there automatic retries on the browser for 5XX status codes?

10

Hi @keith,
The detailed logs had rolled out, so there wasn’t a way to definitively say why the 503 was happening. Generally this happens because the remote storage throws an exception when we are trying to POST. It likely wouldn’t be a rate limit unless the user was uploading with multiple browsers at the same time (the limit is fairly high). It is possible that Google was throwing these exceptions, itself.

The web client is not meant to be a robust upload tool. It is really meant for small or one-off uploads. All sustained or bulk uploads should be done via the desktop client, which has substantial logic around syncing, handling exceptions, backing off on rate limits, retrying, etc. That would be my strong recommendation for this user.