Table of Contents

Use the Micronaut Object Storage API to store files in Amazon S3

Learn how to upload and retrieve files from Amazon S3 using the Micronaut Object Storage API

Authors: Álvaro Sánchez-Mariscal

Micronaut Version: 4.6.3

1. Getting Started

In this guide, we will create a Micronaut application written in Java.

It will be a microservice to store, retrieve and delete profile pictures for users.

2. What you will need

To complete this guide, you will need the following:

  • Some time on your hands

  • A decent text editor or IDE (e.g. IntelliJ IDEA)

  • JDK 17 or greater installed with JAVA_HOME configured appropriately

  • An AWS account with:

    • An IAM user with permissions to create and manage AWS S3 buckets.

    • The AWS CLI configured to use the IAM user above.

3. Amazon Web Services (AWS)

If you don’t have one already, create an AWS Account.

3.1. AWS CLI

Follow the AWS documentation for installing or updating the latest version of the AWS CLI.

3.2. Administrator IAM user

Instead of using your AWS root account, it is recommended that you use an IAM administrative user. If you don’t have one already, follow the steps below to create one:

aws iam create-group --group-name Administrators
aws iam create-user --user-name Administrator
aws iam add-user-to-group --user-name Administrator --group-name Administrators
aws iam attach-group-policy --group-name Administrators --policy-arn $(aws iam list-policies --query 'Policies[?PolicyName==`AdministratorAccess`].{ARN:Arn}' --output text)
aws iam create-access-key --user-name Administrator

Then, run aws configure to configure your AWS CLI to use the Administrator IAM user just created.

4. Solution

We recommend that you follow the instructions in the next sections and create the application step by step. However, you can go right to the completed example.

5. Writing the Application

Create an application using the Micronaut Command Line Interface or with Micronaut Launch.

mn create-app example.micronaut.micronautguide \
    --features=object-storage-aws,graalvm \
    --build=gradle \
    --lang=java \
    --test=junit
If you don’t specify the --build argument, Gradle with the Kotlin DSL is used as the build tool.
If you don’t specify the --lang argument, Java is used as the language.
If you don’t specify the --test argument, JUnit is used for Java and Kotlin, and Spock is used for Groovy.

The previous command creates a Micronaut application with the default package example.micronaut in a directory named micronautguide.

If you use Micronaut Launch, select Micronaut Application as application type and add object-storage-aws, and graalvm features.

If you have an existing Micronaut application and want to add the functionality described here, you can view the dependency and configuration changes from the specified features, and apply those changes to your application.

Diff

Setup IntelliJ IDEA to develop Micronaut Applications.

6. Create an Amazon S3 bucket

First, we are going to create an Amazon S3 bucket using the aws CLI:

aws s3api create-bucket --bucket micronaut-guide-object-storage --region=us-east-1
Bucket names must be unique across all AWS accounts.

If you want to create the bucket in a region other than us-east-1, you need additional parameters. For example, to create the bucket in the eu-west-3 region:

aws s3api create-bucket \
          --bucket micronaut-guide-object-storage \
          --region eu-west-3 \
          --create-bucket-configuration LocationConstraint=eu-west-3

Then, configure the bucket in application.properties:

src/main/resources/application.properties
micronaut.object-storage.aws.default.bucket=micronaut-guide-object-storage

7. Controller API

Let’s define an interface with the endpoints of the profile pictures microservice:

src/main/java/example/micronaut/ProfilePicturesApi.java
public interface ProfilePicturesApi {

    @Post(uri = "/{userId}", consumes = MediaType.MULTIPART_FORM_DATA) (1)
    HttpResponse upload(CompletedFileUpload fileUpload, String userId, HttpRequest<?> request);

    @Get("/{userId}") (2)
    Optional<HttpResponse<StreamedFile>> download(String userId);

    @Status(HttpStatus.NO_CONTENT) (3)
    @Delete("/{userId}") (4)
    void delete(String userId);
}
1 The @Post annotation maps the method to an HTTP POST request.
2 The @Get annotation maps the method to an HTTP GET request.
3 You can return void in your controller’s method and specify the HTTP status code via the @Status annotation.
4 The @Delete annotation maps the delete method to an HTTP Delete request on /{userId}.

And then, an implementation with the required dependencies:

src/main/java/example/micronaut/ProfilePicturesController.java
@Controller(ProfilePicturesController.PREFIX) (1)
@ExecuteOn(TaskExecutors.BLOCKING) (2)
public class ProfilePicturesController implements ProfilePicturesApi {

    static final String PREFIX = "/pictures";

    private final AwsS3Operations objectStorage; (3)
    private final HttpHostResolver httpHostResolver; (4)

    public ProfilePicturesController(AwsS3Operations objectStorage, HttpHostResolver httpHostResolver) {
        this.objectStorage = objectStorage;
        this.httpHostResolver = httpHostResolver;
    }

}
1 The class is defined as a controller with the @Controller annotation mapped to the path /pictures.
2 It is critical that any blocking I/O operations (such as fetching the data from the database) are offloaded to a separate thread pool that does not block the Event loop.
3 AwsS3Operations is the cloud-specific interface for using object storage.
4 HttpHostResolver allows you to resolve the host for an HTTP

7.1. Upload endpoint

Implement the upload endpoint by receiving the file from the HTTP client via CompletedFileUpload, and the userId path parameter. Upload it to S3 using AwsS3Operations, and then return its ETag in an HTTP response header to the client:

src/main/java/example/micronaut/ProfilePicturesController.java
@Override
public HttpResponse<?> upload(CompletedFileUpload fileUpload, String userId, HttpRequest<?> request) {
    String key = buildKey(userId); (1)
    UploadRequest objectStorageUpload = UploadRequest.fromCompletedFileUpload(fileUpload, key); (2)
    UploadResponse<PutObjectResponse> response = objectStorage.upload(objectStorageUpload, builder -> {  (3)
        builder.acl(ObjectCannedACL.PUBLIC_READ); (4)
    });

    return HttpResponse
            .created(location(request, userId)) (5)
            .header(HttpHeaders.ETAG, response.getETag()); (6)
}

private static String buildKey(String userId) {
    return userId + ".jpg";
}

private URI location(HttpRequest<?> request, String userId) {
    return UriBuilder.of(httpHostResolver.resolve(request))
            .path(PREFIX)
            .path(userId)
            .build();
}
1 The key represents the path under which the file will be stored.
2 You can use any of the UploadRequest static methods to build an upload request.
3 The upload operation returns an UploadResponse, which wraps the cloud-specific SDK response
4 The upload can be customised by using Amazon’s PutObjectRequest.Builder methods.
5 We return the absolute URL of the resource in the Location header.
6 The response object contains some common properties for all cloud vendors, such as the ETag, that is sent in a header to the client.

7.2. Download endpoint

For the download endpoint, simply retrieve the entry from the expected key, and transform it into an StreamedFile:

src/main/java/example/micronaut/ProfilePicturesController.java
@Override
public Optional<HttpResponse<StreamedFile>> download(String userId) {
    String key = buildKey(userId);
    return objectStorage.retrieve(key) (1)
            .map(ProfilePicturesController::buildStreamedFile); (2)
}

private static HttpResponse<StreamedFile> buildStreamedFile(AwsS3ObjectStorageEntry entry) {
    GetObjectResponse nativeEntry = entry.getNativeEntry();
    MediaType mediaType = MediaType.of(nativeEntry.contentType());
    StreamedFile file = new StreamedFile(entry.getInputStream(), mediaType).attach(entry.getKey());
    MutableHttpResponse<Object> httpResponse = HttpResponse.ok()
            .header(HttpHeaders.ETAG, nativeEntry.eTag()); (3)
    file.process(httpResponse);
    return httpResponse.body(file);
}
1 The retrieve operation returns an ObjectStorageEntry, in this case an AwsS3ObjectStorageEntry, which allows accessing the AWS-specific GetObjectResponse object in case you need more details about the object.
2 We transform the AwsS3ObjectStorageEntry into an HttpResponse<StreamedFile>.
3 The response contains not only the file, but also an ETag header.
The HTTP client could have used the ETag from the upload operation and send it in a If-None-Match header in the download request to implement caching, which then would have been to be implemented in the download endpoint. But this is beyond the scope of this guide.

7.3. Delete endpoint

For the delete endpoint, all we have to do is invoke the delete method with the expected key:

src/main/java/example/micronaut/ProfilePicturesController.java
@Override
public void delete(String userId) {
    String key = buildKey(userId);
    objectStorage.delete(key);
}

8. Running the Application

To run the application, use the ./gradlew run command, which starts the application on port 8080.

9. Generate a Micronaut Application Native Executable with GraalVM

We will use GraalVM, the polyglot embeddable virtual machine, to generate a native executable of our Micronaut application.

Compiling native executables ahead of time with GraalVM improves startup time and reduces the memory footprint of JVM-based applications.

Only Java and Kotlin projects support using GraalVM’s native-image tool. Groovy relies heavily on reflection, which is only partially supported by GraalVM.

9.1. GraalVM installation

The easiest way to install GraalVM on Linux or Mac is to use SDKMan.io.

Java 17
sdk install java 17.0.12-graal
Java 17
sdk use java 17.0.12-graal

For installation on Windows, or for manual installation on Linux or Mac, see the GraalVM Getting Started documentation.

The previous command installs Oracle GraalVM, which is free to use in production and free to redistribute, at no cost, under the GraalVM Free Terms and Conditions.

Alternatively, you can use the GraalVM Community Edition:

Java 17
sdk install java 17.0.9-graalce
Java 17
sdk use java 17.0.9-graalce

9.2. Native executable generation

To generate a native executable using Gradle, run:

./gradlew nativeCompile

The native executable is created in build/native/nativeCompile directory and can be run with build/native/nativeCompile/micronautguide.

It is possible to customize the name of the native executable or pass additional parameters to GraalVM:

build.gradle
graalvmNative {
    binaries {
        main {
            imageName.set('mn-graalvm-application') (1)
            buildArgs.add('--verbose') (2)
        }
    }
}
1 The native executable name will now be mn-graalvm-application
2 It is possible to pass extra arguments to build the native executable

10. Testing

We can test our application from the command line.

10.1. Uploading a profile picture

If you want to upload a file larger than 1MB, you need to configure:

src/main/resources/application.properties
# 20 * 1024 * 1024 = 20MB
micronaut.server.multipart.max-file-size=20971520

Assuming you have locally a profile picture in a profile.jpg file, you can send it to your application with:

$ curl -i -F 'fileUpload=@profile.jpg' http://localhost:8080/pictures/alvaro

HTTP/1.1 201 Created
location: http://localhost:8080/pictures/alvaro
ETag: "617cb82e296e153c29b34cccf7af0908"
date: Wed, 14 Sep 2022 12:50:30 GMT
connection: keep-alive
transfer-encoding: chunked

Note the Location and ETag headers.

We can use the aws CLI to verify that the file has actually been uploaded to an S3 bucket:

aws s3 ls s3://micronaut-guide-object-storage/

10.2. Download a profile picture

curl http://localhost:8080/pictures/alvaro -O -J

The file will be saved as alvaro.jpg since our download endpoint includes a Content-Disposition: attachment header. Open it to check that it is actually the same image as profile.jpg.

10.3. Delete a profile picture

aws s3 ls s3://micronaut-guide-object-storage/

10.4. Cleanup

Once you are done with this guide, you can remove the bucket from S3 to not leave stale resources:

aws s3api delete-bucket --bucket micronaut-guide-object-storage --region eu-west-3

11. Next Steps

12. License

All guides are released with an Apache license 2.0 license for the code and a Creative Commons Attribution 4.0 license for the writing and media (images…​).