I am a big fan of AWS Lambda; a cloud managed function based micro-service by Amazon.
The first in a series of articles is about how to enable indexing of objects landing in S3 without any other action from the publisher (except of course landing this file in s3).
The code of this article is at is https://github.com/milindparikh/storiesofaws/tree/master/spooky-action-at-a-distance
As a analyst, I want to search files landing in S3 by it's content as soon as the file lands in S3.
The analyst has access to CloudSearch. The publisher has access to S3 bucket. S3 itself can notify certain objects on a file getting inserted into the bucket. But no way to connect the two. The architectural use case to to illustrate the use of Lambda for the uber user story.Lambda is NOT appropriate in all use cases and needs additional components.
AWS Lambda to the rescue.
As shown in Figure 2, an AWS Lambda function can serve as the essential glue between the storage and search.
It gives the illusion of having search results automatically reflect the puts in the S3 bucket.
An AWS Lambda function can only function in context of role given to it. Also only specific roles can invoke a Lambda function.... etc.... all of this has be configured either in the console or SDK or.. you might just want to take a peek at code.
Both the core function and the role generation is template driven in the code. This means that you can change the core function to suit the needs of your particular search config.
My CloudSearch Domain
Figure 3 shows my cloud search domain config. It indices two fields (name and desc) and ignores (notice the * field) the rest.
The first in a series of articles is about how to enable indexing of objects landing in S3 without any other action from the publisher (except of course landing this file in s3).
The code of this article is at is https://github.com/milindparikh/storiesofaws/tree/master/spooky-action-at-a-distance
THE UBER USER STORY
As a analyst, I want to search files landing in S3 by it's content as soon as the file lands in S3.
AN ARCHITECTURAL PROTOTYPE
 |
| Figure 1 |
The analyst has access to CloudSearch. The publisher has access to S3 bucket. S3 itself can notify certain objects on a file getting inserted into the bucket. But no way to connect the two. The architectural use case to to illustrate the use of Lambda for the uber user story.Lambda is NOT appropriate in all use cases and needs additional components.
AWS Lambda to the rescue.
 |
| Figure 2 |
As shown in Figure 2, an AWS Lambda function can serve as the essential glue between the storage and search.
It gives the illusion of having search results automatically reflect the puts in the S3 bucket.
THE GORY DETAILS
Security
An AWS Lambda function can only function in context of role given to it. Also only specific roles can invoke a Lambda function.... etc.... all of this has be configured either in the console or SDK or.. you might just want to take a peek at code.
The Core Function
Both the core function and the role generation is template driven in the code. This means that you can change the core function to suit the needs of your particular search config.
My CloudSearch Domain
 |
| Figure 3 |
