[FOLIO-2340] vagrant downloads may be very slow in Europe Created: 04/Nov/19 Updated: 03/Jun/20 Resolved: 07/Nov/19 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Bug | Priority: | TBD |
| Reporter: | Zak Burke | Assignee: | Wayne Schneider |
| Resolution: | Done | Votes: | 0 |
| Labels: | devops, platform-backlog | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
| Sprint: | CP: sprint 76 |
| Story Points: | 2 |
| Development Team: | Core: Platform |
| Description |
|
Very slow download speeds for Folio's Vagrant boxes are a rare but recurring problem. Users in France, Italy, and eastern Europe have reported download speeds on the order of ~160k, resulting in expected download times of 8-10 hours. On the east coast of the US, download speeds are > 6000k, resulting in expected download times of ~15 minutes. This has been confirmed on separate occasions with Claire Hernandez, Matteo Pascucci, and Kostyantyn Khodarev. |
| Comments |
| Comment by Wayne Schneider [ 04/Nov/19 ] |
|
OK, some quick notes from 10 minutes of research and thinking:
|
| Comment by Peter Murray [ 04/Nov/19 ] |
|
Yes, we are watching costs closely, but need to balance that with people being effective. I'm not sure about S3 Transfer Acceleration, but CloudFront might be another option as well. That'll cost significantly more, though, as well. |
| Comment by Wayne Schneider [ 04/Nov/19 ] |
|
S3 transfer acceleration is essentially CloudFront, as I understand it. Current transfer cost out of S3 are (assuming we are under 10TB/month) $0.09/GB. Transfer acceleration would add $0.04/GB, but only for transfers that require it. That is, if AWS determines that regular S3 transfer would be just as fast or better for a particular transfer, we would pay only the base rate. |
| Comment by Peter Murray [ 04/Nov/19 ] |
|
Did some Googling. The "Vagrant box downloads extremely slow" thread on GitHub was interesting, particularly with the last comment about using the operating system binaries rather than the embedded binaries. That might be worth a try. A few comments above there was a reference to putting the URL at the end of the Vagrant line; something like ```vagrant box add "bento/ubuntu-16.04" https://app.vagrantup.com/bento/boxes/ubuntu-16.04/versions/201802.02.0/providers/virtualbox.box``` (except using the FOLIO-specific names and URLs, of course). I'm going to assume people are using Vagrant versions after 2.0.3, so this comment about Vagrant seeing the redirect in the display. This blog post was insightful as well: Download speed : vagrant init vs wget vs aria2 — Sanjeev Nithyanandam's Blog |
| Comment by Wayne Schneider [ 05/Nov/19 ] |
|
Assigning to Wayne Schneider per Jakub Skoczen's request. Story points set to "2" – if the solution is simple (e.g. S3 transfer acceleration), then great, if not it could be arbitrarily complex |
| Comment by Wayne Schneider [ 05/Nov/19 ] |
|
I have turned on S3 transfer acceleration on the folio-vagrant bucket for testing purposes. I created a test box on Vagrant Cloud that points to the accelerated endpoint (the "normal" boxes still point to the standard transfer endpoint, so should not be accelerated). I reached out to Matteo Pascucci to help with testing. |
| Comment by Wayne Schneider [ 06/Nov/19 ] |
|
In testing with Matteo, a vagrant box add from the accelerated endpoint reported about 4100kbps transfer rate, while the same command from the standard endpoint continues to transfer at 150kbps or less. That's a pretty dramatic improvement for a simple solution. I've asked OLF for permission to pursue this. If we go down this route, all it will require is a minor update to the variables in the folio-blackbox Jenkins job in folio-infrastructure, I believe. |
| Comment by Peter Murray [ 06/Nov/19 ] |
|
I created an AWS budget to warn if S3 Transfer Acceleration exceeds $100/month. If it is bigger than that, I think we should take this request to Technical Council for approval. cc: Mike Gorrell. |
| Comment by Peter Murray [ 07/Nov/19 ] |
|
The costs for this are, so far, negligible. (Look for the Amazon S3 accelerated data transfer out lines in this screenshot.) This is month-to-date. |
| Comment by Wayne Schneider [ 07/Nov/19 ] |
|
I have updated the packer build to write the accelerated URL into the box metadata. All future box builds will use the accelerated URL (but not all transfers will happen at the accelerated transfer price, at least according to Amazon's docs – supposedly if regular transfer is just as fast we will still get the regular transfer rate). Thanks, Peter Murray! Let me know if this gets out of hand. |
| Comment by Peter Murray [ 08/Nov/19 ] |
|
This continues to have a negligible impact in the broader scheme of things: |
| Comment by Claire Hernandez [ 12/Nov/19 ] |
|
Hi, This morning on snapshot-core: Rate: 4355k/s Thanks a lot Claire. |
| Comment by Peter Murray [ 12/Nov/19 ] |
|
Added cost continues to be negligible. Turning this on was a good call, Wayne; thanks! |
| Comment by Peter Murray [ 21/Nov/19 ] |
|
Continues to be negligible: |
| Comment by Peter Murray [ 02/Dec/19 ] |
|
Yup – maybe $25-$35 impact for the month. Let's keep this on. |
| Comment by Wayne Schneider [ 02/Dec/19 ] |
|
Thanks for following up, Peter Murray! |