[FOLIO-3418] clean up old Vagrant images from S3 Created: 17/Feb/22  Updated: 05/May/23

Status: Open
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Story Priority: P3
Reporter: Jakub Skoczen Assignee: Wayne Schneider
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Sprint: DevOps Sprint 135, DevOps Sprint 136, Kitfox: sprint 165, DevOps Sprint 134
Development Team: FOLIO DevOps
RCA Group: TBD

 Description   
  • Clean up old images from S3 buckets.
  • Clean up old Vagrant cloud images
  • Clean up the build scripts (reference to deprecated images e.g "testing")
  • Clean up the documentation

Review other existing S3 buckets to decide if they should be shutdown.



 Comments   
Comment by Marc Johnson [ 03/Mar/22 ]

Jakub Skoczen Wayne Schneider

I'm not sure if this is the correct issue. Jakub Skoczen asked me to comment on the issue assigned to Wayne Schneider with feedback about the retirement of the testing vagrant images.

In the Technical Council yesterday, it was mentioned that the snapshot-backend vagrant image is intended to be equivalent to the retired testing-backend image.

I passed this on to folks who are using the testing-backend image.

When a developer tried to use the snapshot-backend image, they got the following error:

The box 'folio/snapshot-backend' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Vagrant Cloud, please verify you're logged in via
`vagrant login`. Also, please double-check the name. The expanded
URL and error message are shown below:

URL: ["https://vagrantcloud.com/folio/snapshot-backend"]
Error: The requested URL returned error: 404 Not Found

Please can you help us find the snapshot-backend images?

Comment by Wayne Schneider [ 03/Mar/22 ]

Sorry for the confusion, Marc Johnson. There is not a snapshot-backend image. The snapshot image backend should be the same as the testing-backend image (though they are constructed slightly differently), because interface dependencies for the backend were always enforced (the main difference between the snapshot and testing scripts is that the testing script used the most recent "snapshot" releases of front-end modules without enforcing interface dependencies).

It's always been a rather confusing situation. This probably doesn't explain it completely, either

Comment by Wayne Schneider [ 03/Mar/22 ]

To be clear, the only difference between the testing-backend and the testing images was that the testing image had a Docker container running nginx hosting a webpack bundle composed of the most recent snapshot releases of front-end modules.

Comment by Marc Johnson [ 03/Mar/22 ]

Wayne Schneider

Thank you for responding quickly.

There is not a snapshot-backend image.

That is good to know.

The snapshot image backend should be the same as the testing-backend image (though they are constructed slightly differently), because interface dependencies for the backend were always enforced (the main difference between the snapshot and testing scripts is that the testing script used the most recent "snapshot" releases of front-end modules without enforcing interface dependencies).

Thanks for explaining that. I don't think folks are concerned about the slightly different construction, except for the side-effect of that which means one of the images only has back end modules and the other has back end and front end.

To be clear, the only difference between the testing-backend and the testing images was that the testing image had a Docker container running nginx hosting a webpack bundle composed of the most recent snapshot releases of front-end modules.

The feedback I've been given is that this adds significantly to the resource usage of the image, especially when it is initially spun up, to the point where machines that can run the back end only one cannot run this one.

Comment by Jakub Skoczen [ 04/Mar/22 ]

Wayne Schneider can you verify this (the res alloc for running with or without the nginx+bundle)? I am surprised that adding a nginx container makes a big difference so we I'd appreciate if you also checked if other factors could make snapshot image more heavy than the testing one (e.g individual container configurations). Thanks.

Comment by Wayne Schneider [ 07/Mar/22 ]

I think the difference in resource allocation is not really about the nginx container, which is pretty small. The difference is that in the testing build, all backend modules were manually enumerated for deployment in a configuration file, rather than being pulled in based on dependency resolution. Some modules were deliberated excluded (e.g mod-ldp), and some modules were being inadvertently excluded.

I think the best solution is to push forward the work for "platform-minimal" ( FOLIO-3253 Closed ) so that there is a smaller set of requirements for a coherent front- and backend system.

We could also look at building a backend-only box with a specified set of modules (for example, only the modules required for authn/z on the backend). If you feel like that would be useful for your team, please raise another ticket and we can look at what your requirements might be.

Comment by Marc Johnson [ 07/Mar/22 ]

Wayne Schneider

I think the difference in resource allocation is not really about the nginx container, which is pretty small.

I don't understand how vagrant images are built.

Are the steps run on the machine running the image, e.g. when a developer starts the image, does it spin up Okapi and deploy the modules and build the bundle for the front end?

I think the best solution is to push forward the work for "platform-minimal" ( FOLIO-3253 Closed ) so that there is a smaller set of requirements for a coherent front- and backend system.

Ok. I don't anticipate that work will progress quickly.

We could also look at building a backend-only box with a specified set of modules (for example, only the modules required for authn/z on the backend). If you feel like that would be useful for your team, please raise another ticket and we can look at what your requirements might be.

I don't think folks are going to do that.

I think the conclusion of this conversation is that the belief that there were equivalent snapshot vagrant images to the removed testing ones turned out not to be the case and those folks using them will need to find other ways.

Comment by Jakub Skoczen [ 07/Mar/22 ]

Marc Johnson The selection of modules in the "testing" backend Vagrant image did not follow any particular plan. It just so happened that some modules were held back from that image while others were simply forgotten. Either way the fact that the "testing" backend had a smaller footprint was a pure chance rather than "by design". It's certainly not something the developers should depend on.

I think "platform-minimal" is something that we should priorize as a project as this would be a foundation for a small footprint Vagrant image that developers could use. If you bring this to the TC I will support any motions to get this work on track.

Comment by Marc Johnson [ 07/Mar/22 ]

Jakub Skoczen

The selection of modules in the "testing" backend Vagrant image did not follow any particular plan. It just so happened that some modules were held back from that image while others were simply forgotten. Either way the fact that the "testing" backend had a smaller footprint was a pure chance rather than "by design". It's certainly not something the developers should depend on.

Ok. Given those images are now retired, it isn't something the folks who were using them are going to be concerned with now.

I think "platform-minimal" is something that we should priorize as a project as this would be a foundation for a small footprint Vagrant image that developers could use. If you bring this to the TC I will support any motions to get this work on track.

As far as I can tell, this initiative has stalled (since the tech leads meeting about it).

I don't think I'm the person to represent or sponsor that initiative.

Comment by Marc Johnson [ 07/Mar/22 ]

Jakub Skoczen

I think "platform-minimal" is something that we should priorize as a project as this would be a foundation for a small footprint Vagrant image that developers could use. If you bring this to the TC I will support any motions to get this work on track.

Though I do appreciate the offer to support this initiative.

I do think it's an important area, especially given the concerns folks have around developer onboarding and community development.

I think it would be great if FOLIO had a cohesive plan for how it wanted to structure both development and production systems (in terms of operations and how they are extended / enhanced).

I don't think I'm the person to lead that initiative, given that I neither use the existing vagrant images, the scratch environments or operate hosted environments.

Generated at Thu Feb 08 23:27:58 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.