[FOLIO-2450] FTP/SFTP servers for the reference environments Created: 07/Feb/20 Updated: 08/Feb/23 Resolved: 20/Mar/20 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | Continuous Integration |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Story | Priority: | P3 |
| Reporter: | Craig McNally | Assignee: | David Crossley |
| Resolution: | Done | Votes: | 0 |
| Labels: | devops, devops-backlog | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||
| Sprint: | DevOps: sprint 84, DevOps: sprint 83 | ||||||||||||
| Story Points: | 5 | ||||||||||||
| Development Team: | FOLIO DevOps | ||||||||||||
| Description |
OverviewThe Acquisitions team will soon start work on features that involve uploading files to an FTP/SFTP server. More specifically, generate a batch voucher and upload it to a configured FTP location w/ the configured credentials. In order to support testing in the reference environments (folio-testing/snapshot/etc.) we'll need to provision an FTP/SFTP server. I'm also aware that somewhere on the roadmap are other features which work with FTP, so this is the first but not the only feature requiring something like this. Examples include EDI/Edifact ordering and invoicing, possibly others too (export?) There are no requirements for a particular directory structure, or specific user credentials to be provisioned. I imagine that there's no real need for long-lived data here as it's only for testing purposes. Periodic aging out or purging of data should be perfectly acceptable for our current needs. At this point the load is expected to be low so something lightweight is probably fine. Acceptance criteria
|
| Comments |
| Comment by Jakub Skoczen [ 10/Feb/20 ] |
|
Waiting on decision which protocol needs to be supported (FTP vs SFTP) |
| Comment by Craig McNally [ 11/Feb/20 ] |
|
Jakub Skoczen both since both protocols will be supported by the code |
| Comment by Ann-Marie Breaux (Inactive) [ 27/Feb/20 ] |
|
Hi Craig McNally I would agree about needing both. Depending on who is on the other end of the connection as sender or recipient, they may have specific requirements about SFTP, while many are fine with regular FTP. It's not an either/or. It's both. I know GOBI runs both types of servers due to differing requirements from the libraries or partners on the other end of the transactions. |
| Comment by John Malconian [ 03/Mar/20 ] |
|
AWS has a managed SFTP service, https://aws.amazon.com/sftp/features/, however, I think it may be overkill for the FOLIO requirements and may be cost prohibitive. Plus it doesn't support FTP. I think the best bet is to provision a t2.micro or t3.micro type instance (or whatever the current free-tier eligible instance type is) and configure FTP and SSH services. Create a dedicated user that is used by the FOLIO app to authenticate to both services but doesn't have any other special permissions. A container can on the kubernetes cluster could also be used, but that may require additional work in order to configure the ingress to support SSH and SFTP. I think the micro ec2 instance is the fastest and cheapest option. |
| Comment by Craig McNally [ 03/Mar/20 ] |
|
Sounds reasonable to me. Something like vsftp (https://security.appspot.com/vsftpd.html) is pretty easy to setup and is probably more than adequate for the FTP side of things. For sftp I think everything you need is probably already on the base AMI (sshd). |
| Comment by Craig McNally [ 03/Mar/20 ] |
|
John Malconian You may be able to find publicly available docker images that do what you need |
| Comment by Jakub Skoczen [ 16/Mar/20 ] |
|
David Crossley : John Malconian says we should use the ci.folio.org domain and verify if the LB can handle FTP/SFTP to save on elastic IPs. |
| Comment by John Malconian [ 17/Mar/20 ] |
|
David Crossley I was able to reallocate a previously provisioned elastic IP so no need to do any load balancing. The permanent IP address associated with the FTP/SFTP instance is 52.7.45.151. You will want to update the 'pasv_address' parameter in vsftpd.conf with this address. Add a DNS entry that points to this IP and you should be all set. |
| Comment by David Crossley [ 17/Mar/20 ] |
|
Thanks John, that part is now sorted. Some documentation is in progress. Getting close now. |
| Comment by David Crossley [ 18/Mar/20 ] |
|
See documentation via pull/543 and the generated document in-development. |
| Comment by David Crossley [ 20/Mar/20 ] |
|
The document is published at https://dev.folio.org/guides/ftp-ci-server/ Also a general reminder to use the "Search" facility to find stuff there. |
| Comment by David Crossley [ 08/Feb/23 ] |
|
Update to assist Jira search: As explained in the documentation: ftp.ci.folio.org and the AWS EC2 instance i-0e6b3d9f00a6cea5f is tagged as folio-ftp |