[FOLIO-2820] SPIKE: Docker Hub download rate limiting Created: 06/Oct/20 Updated: 03/Nov/20 Resolved: 02/Nov/20 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Story | Priority: | P2 |
| Reporter: | Wayne Schneider | Assignee: | John Malconian |
| Resolution: | Done | Votes: | 0 |
| Labels: | devops-backlog | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||||||||||||||||||||||
| Sprint: | DevOps: Sprint 99, DevOps: Sprint 100 | ||||||||||||||||||||||||||||||||
| Development Team: | FOLIO DevOps | ||||||||||||||||||||||||||||||||
| Description |
|
Starting this month, with full enforcement by Nov 1, Docker Hub will begin enforcing download rate limits: https://docs.docker.com/docker-hub/download-rate-limit The practical upshot:
Since standing up a full FOLIO environment involves about 60 docker pull requests, this has imminent implications for our CI builds. In addition, other implementers (hosting providers, etc.) will start to run into issues. Things to consider:
|
| Comments |
| Comment by John Malconian [ 06/Oct/20 ] |
|
One idea may be to use the FOLIO Nexus repository as a caching Docker proxy to Docker Hub similar to the way it's used for NPM and Maven Central. This potential option is really only a viable for FOLIO project CI infrastructure and not for general use by the community, however. I'm not sure there is a way to make the proxy transparent, but we can look into that. Otherwise, we'd have to update various build pipelines to use the Nexus Docker proxy registry. In this scenario, the authenticated Docker user would be Nexus. |
| Comment by Jakub Skoczen [ 13/Oct/20 ] |
|
John Malconian Wayne Schneider we've discussed that the least intrusive way is to wait for
|
| Comment by Peter Murray [ 13/Oct/20 ] |
|
I'm pursuing two tracks. First is to get us recognized as an open source project (
|
| Comment by Jakub Skoczen [ 20/Oct/20 ] |
|
with
1. update ansible to make use of the new Okapi feature Stanislav Miroshnichenko Any ideas about the 3. thing? |
| Comment by Stanislav Miroshnichenko [ 20/Oct/20 ] |
|
Jakub Skoczen Rancher can store auth. credentials for Docker Registry, such as DockerHub. But these credentials will be available for team's members in Rancher for reading. |
| Comment by Peter Murray [ 02/Nov/20 ] |
|
Note that this may turn out to be a non-issue, as the `folioorg` and `folioci` repos would be exempted from the rate limits when the project is recognized as an open source project. See this comment on FOLIO-2722 for more details. |
| Comment by Jakub Skoczen [ 03/Nov/20 ] |
|
Stanislav Miroshnichenko Wayne Schneider We talked about using the FOLIO project Nexus proxy for pulling images from DockerHub. Wayne is going to provide Nexus access credentials to you. |
| Comment by Jakub Skoczen [ 03/Nov/20 ] |
|
Nexus repo for pushing scratch env images: docker.dev.folio.org |