You might think of Java as outdated. It is, indeed, 27 years old, which is the equivalent of one century old in “IT years”. Its younger, fresher, hipper siblings — such as Scala, Kotlin, Closure, or Groovy — might be more dynamic and easier to use. But don’t discard Java just yet!

Andrei Micu, Senior Scala Developer at Levi9, has noticed that Java is evolving and using its younger siblings for inspiration and improvement. Andrei talks here about two projects that intend to evolve the JDK, with their associated JEPs (JDK Enhancement Proposals): Project Amber and Project Valhalla.

Let’s see some instances where Java is a bit behind its siblings and how syntax sugar and standardization simplifies and makes the code more efficient.

POJOS (Plain Old Java Objects) make data encapsulation a bit difficult. They needed a lot of writing but other languages such as Scala and Kotlin came up with features that standardized and simplified the writing. The same output could be achieved with a case class for Scala or data class for Kotlin. And this is what Project Amber tries to change.

Project Amber started with JDK15 and did some syntax sugar by adding the concept of records that brought in the same feature from Scala or Kotlin.

Another interesting change brought on by Project Amber is pattern matching. Old Java is not very flexible with switches, but Scala is way ahead. For example, in the second Scala example below — you can switch in the same object for lists, for components of lists and you can extract from them.

The concept of pattern matching for instanceof was added in the first release of JDK14. Previously, when you checked if an object was an instance of a class, an extra line was necessary for the class to be cast to its checked type. But since JDK14 you can simply write the name of the variable after the class and you don’t need that extra line. This is a small, neat improvement.

JDK17 brought on pattern matching in a switch. If you want to check an object for its class type, now it’s possible in switch statements too.

JDK18 came along with increased expressiveness. Not only can you now check classes in switch statements, but you can also check for particular values and classes in the same switch. Guards and nulls can also be added.

In the future, we will also have record patterns. Previously, if we got a record at this point you had to write several lines of code to access its members, but now you can do an instanceof point with the name of the internal fields and it’s going to extract them for convenient use, afterwards.

In switches you can do the same for cases. This sort of expressiveness is going to deconstruct that point, which is something pretty new in Java, so that’s why it probably takes more time to cook.

We are now in the territory of primitive types. This is the focus of Project Valhalla. In Java, primitive types are enough to model almost anything. But anything that’s not on the primitive type list is derived from Object, and this is quite an issue. Scala and Kotlin have a different approach to this. Both languages use just one class and the compiler does the improvements for you.

However, Java does not have a class for primitive types, making it difficult to work with them seamlessly.

But this may change and primitive types might get a class. This class will be marked with a `Q` prefix in the JVM internals to signal it’s a primitive type.

Java wants to offer the possibility of making objects as primitive types. The first step is to give the possibility of declaring objects that don’t have identity.

But what does “having identity” mean? Long story short, when an object has identity, two instances of the same value are different. This is not the case for primitive types and this is why we say that primitive types do not have identity.

Furthermore, objects that don’t have identity are stored in the stack, while objects with identity are stored in the heap.

Value objects (or classes) don’t have identity. Let’s look at how they are declared.

While the outline of the definition is still work in progress, there are some definition rules that have been already established. One is that the class should be implicitly final. The fields of the class are implicitly final and no constructor can call super. Also, value objects can have reference cycles, which means they can have references to another object of the same type.

What do we get in return? The objects are put on the stack and the equality will hold between two references to the same value..

A primitive class is like a value class with the additional restriction that no field can be of the declaring class, so we don’t have cycles like we have for value objects.

What do we get in return? We get almost the same performance as primitive-types. They do not allow nulls. Some might say this is not a benefit, but those who had null-pointer exceptions might think otherwise.

While not yet official, Project Valhalla does drop a hint about tackling reified generics, in which they may also take a hint from Scala and Kotlin.

What is the issue in Java right now? Well, when generic type objects are passed around, the information about the inner types is lost at runtime. This is called `type erasure` and it is a drawback of the JVM.

There is a workaround in Java, that requires you to send the class as a separate parameter. Definitely not elegant! But Kotlin and Scala have a better way of dealing with this.

Kotlin deals with this with a neat feature called “reified generics”. If you declare the function inline — meaning that the function content is copied at the call site — then you can have access to the inner types of the generics. When you write reified, it adds in the back an extra parameter which is the class of the generic type parameter. This way, you don’t have to write extra parameters.

Scala works around this in a different way. It makes use of implicit parameters to leverage the compiler’s ability to pass the class information.

Do these Java improvements seem significant to you? Andrei Micu stresses that standardization is not easy. Java’s little siblings might be a bit lighter on standardization, which allows them to develop features faster. Java, however, is so widely used, that keeping backward and forward compatibility is crucial. Sometimes, this means holding back on innovation, in order to keep all of its users happy.

“Let’s keep in mind Java is running a marathon”, says Andrei Micu. Its younger sibling languages, with more spring in their step, will also benefit from JVM improvements and new features. It’s a win-win.

130 years ago, when our client first started, a cloud may have been a bad omen. It could have meant the impending approach of a storm preventing its ships from reaching their destination. Today? The cloud is what helps our clients make the best decisions, based on hard, consistent, relevant data. One of these vessels shipping BI services through the cloud is called Levi9, with a team helmed by Carmen Girigan, Senior Data Engineer, and Eliza Enache, Medior Data Engineer.

If a business founded in 1890 is still thriving today, it must mean that it has adapted over and over again. More recently, this meant adopting a cloud-based data solution with the help of Levi9 and other cloud providers. Vroon, an international company with a vast tradition in naval transport, was looking for ways to optimize their data solution, in order to make better decisions. We picked up a legacy on-premises solution and transformed it into a personalized, highly-scalable Single Source of Truth in the cloud.

The main challenge was picking up a complex solution using external data sources, requiring maintenance and development on multiple modules, and offering limited aggregated results. To learn how the business was doing, Vroon business users had to look into multiple sources.

However, data is mostly useless unless it becomes information. But what does this mean?

“You might look at Data as the input that your brain perceives through the senses”, says Eliza Enache, one of the Levi9 “sailors”. “The brain then creates connections between these data points by applying certain logical processes (filtering, algorithms, etc.). The connections between data points (and connections of connections) might be called Information.”

“Raw data is useful only when it is organized and interpreted in such a way that it helps in decision making, becoming information” completes Carmen Girigan, also a member of the Levi9 team.

After working with Levi9, Vroon users interrogate one service and get the answers they need.

The Vroon business had been getting information using on-site Microsoft technologies, relying heavily on SQL Servers. Data was coming in from various siloed sources such as accounting, procurement, invoicing, customers, vessel activities, incident reporting, and many others.

Data were extracted, transformed, and loaded into the warehouse, via Microsoft SQL Server Integration Services, as well as custom applications and SQL Server stored procedures. On top of the data marts, there were 5 analytical databases with 11 multidimensional OLAP cubes, developed with Microsoft SQL Server Analysis Services. These cubes would then be accessed via the reporting tool, for reports, dashboards, and analysis.

The cloud conversion started in 2020. The new solution was based on Microsoft Azure technologies, in order to streamline extract-load-transform (ELT) processes and maintenance, by using a gradual approach that would allow evaluation and constant adjustment to new situations, as well as high availability for the users’ new requirements.

Now, data is extracted from its sources via Azure Data Factory and loaded into an Azure Data Lake container. It then goes through a transformation process using Azure Databricks or Azure Synapse Analytics. Afterward, it is loaded into an Azure Synapse Analytics data warehouse, based on which a tabular model is developed with Visual Studio and later deployed to Azure Analysis Services. Power BI is used for data presentation and user self-service report creation.

The result was the transformation into a modern, cloud-based, and highly-scalable solution, that provides information in a unified way: a Single Source of Truth for multiple business areas.

Having a single source of truth is essential for enabling good business decisions. “Everybody sees the same figures, at the same time, and it’s more reliable” Carmen explains. “Long story short, you avoid conflicting ‘versions of truth’ when different users analyze data exported at different points in time”, adds Eliza.

Whenever thinking of a ship, one image inevitably comes to mind: sailors cleaning the deck, almost to the point of obsession. In the 18th century, this was done to keep the crew healthy.

Today’s Levi9 data engineers have a similar attitude towards data: it should be maintained clean, to avoid the risk of infecting the budget, reports, and any meaningful analysis.

“In the cloud, resources are much easier to maintain and scale — but we do have to pay attention to costs. There is no point in having duplicated or unutilized data, or transformations, in the cloud”, explains Carmen. Eliza adds that there is much more than that. “During the testing phase, we noticed that redundant or obsolete data led to improper analysis and estimation, and diversion from the ‘main track’ — which in turn led to an increase in time and costs, while also blocking further developments. Some of these redundancies made us go back through the whole logic.”

Those who don’t empathize with nautical experience might compare data cleansing with their wardrobes: “Just think of that closet full of clothes you don’t like to wear, things you’ve kept for years only cluttering your space and making for a hefty baggage” says Eliza.

Part of Vroon’s legacy system is still running and is in the process of being transformed. Eliza explains why sometimes this is a better solution: “We soon realized that converting and rewriting the whole logic, from old technologies into Azure, would take a lot of time – thus stalling any further development initiative. As such, we devised a temporary workaround: part of the data is directly brought to and processed in Azure, while another part is brought from the on-premises data warehouse, and further integrated into the unified model.”

“But we won’t do everything at once. First, we’ll do a pilot project”, adds Carmen. Just like sending a sentinel ship before moving the entire fleet, a pilot provides insight into what the job entails.

All in all, the project continues to be a learning opportunity for the team. All the Levi9 Vroon sailors have at least one Azure certification, as the department and company strategy aligned with the work. Carmen Girigan and Eliza Enache stress that they owe much of their learning to the other half of the initial engineering team that had established the foundations on which they continued to build in the cloud.

With the new pilot project, the adventure continues. May the team have fair winds and following seas!

Imagine yourself in a run-down house. Make it a big one. Huge. Some walls are still standing, but barely. Some rooms are just a pile of bricks lying in the middle of the floor. There are secret passageways connecting random chambers, almost like a labyrinth. Did you make the house gigantic in your mind? Go on, make it even bigger. A quirky, dysfunctional, enormous castle.

Here is your challenge: you need to rebuild the castle on another land. In its new version, the building should regain its splendor, while guests and inhabitants alike should be able to find their way seamlessly. Oh, and if you fail, even slightly, even by a fraction, all the inhabitants of the palace lose their livelihood.

Proceed.

This is the magnitude of the challenge that software architect Sebastian Gavril found himself in, when he was approached to migrate a fintech’s core databases to the cloud.

Tinka, a Dutch fintech company, already had some of its services in the cloud, such as microservices, authentication systems, message brokers, CI/CD, and monitoring tools. But the core services, related to the actual financial processes, remained on-premise, part of the so-called legacy system: the virtual machines, databases, the Enterprise Service Bus, and workflows.

The job entailed moving from on-premise into the cloud 30+Tb of data, split in over 30 Oracle database instances across multiple Exadata machines — mini data centers that can be rented out — and 7 environments.

In other words, the core services were still in the “old castle” and needed to be moved. All those discarded bricks, secret passageways — they were actually scattered databases on old servers, connected among each other in a complex network, being used by various applications, microservices, or monitoring tools.

So how does this story unfold?

This job needed an architect. A software architect. “Normally, an architect first needs to understand his customers, their needs and desires, constraints and opportunities. Then he translates that into a formal language that can be used by engineers to construct the building his customers desire. And during the construction phase, the architect is always present to make sure the plan is followed and change requests from clients are included with minimal impact on the integrity of the building.”

This is what Sebastian did. He needed to have an ensemble view of the challenge, decide what’s needed, and get the team together. They were: Simona Gavrilescu (Delivery Manager), Ancuta Costan (Test Lead), and Monica Coros (DevOps Engineer), as well as employees of Tinka.

At Levi9, the team is defined together with the customer, with the aim of enhancing collaboration and working together as one team.

Moving core databases is an extremely complex project. When you move databases, you don’t just move the data, but you need to make sure that all the critical systems will function correctly.

Plus, the old castle — the legacy systems — were responsible for a large part of Tinka’s income. This is true for most companies.

While looking to achieve a positive impact for the clients, one of Levi9’s goals of this project was to have no negative effect on the Tinka customers. Customers were not supposed to notice anything. However, if things did go wrong, the company could have lost or corrupted customer data.

So where was this new land where the castle was being built? The cloud.

Until recently, “the cloud” actually meant just a couple of key players, such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform. Nowadays, any large, established company starts to develop its own cloud: Oracle Cloud, IBM Cloud, Digital Ocean, Heroku, Alibaba. To attract customers, many offers features such as niche-specific services, better integration between cloud and legacy systems, and migration tools.

The chosen solution was the RDS – Relational Database Services, the database service offered by AWS, mainly because part of Tinka’s infrastructure was already in AWS. After migration, AWS would be the one keeping up with maintenance and updates and many more as part of the package.

“It was imperative to understand the complexity, which I saw on two levels”, says Sebastian.

The first level of complexity is that of the database. In this case, the team used a tool provided by the AWS Data Migration Service (DMS) of AWS called the Schema conversion tool. “When you move a database from one place to another, this tool generates a scheme that shows the objects, triggers, procedures, and packages. It also shows which ones can be converted automatically and where manual changes need to be made, through simple, medium, or complex actions. This helps us understand what database-specific effort we need” explains software architect Sebastian Gavril.

The second level of complexity is that of the “landscape”. That means everything that calls on the database. The landscape was very rich. From old, legacy servers – that nobody really understood, to modern microservices, with everything in between: ESB, custom jobs, etc.

It helps us understand where the problems are. What we need to change: connection strings, drivers, maybe even code that used specific functionality in Oracle Enterprise Edition (an edition we wanted to move away from).

So what does moving the castle entail? Well, when planned systematically, things look pretty straightforward: you need to move the bricks and you need to redesign the rooms. In data transfer language this means that you need to take into consideration two aspects: the applications and the databases.

On the applications side, there are some conversions to be made: for example update libraries, change versions, rewrite some code, and make sure the configurations are in order.

The databases also need to be converted, either by using tools such as the schema conversion tool or manually. For some, you can find a procedure. For others, they need to be built from the ground up.

For migrating the actual data, just like renting a moving truck, AWS has a Database Migration Service offer. You can rent a machine, point to the “departure” address, set an “arrival” address, and up you go. It can take minutes, hours, days. Or weeks, in one of our cases.

Every project starts with “it’s not much”. “It’s practically done”. Sebastian Gavril knows to cringe at this one. The first migration project was started 1 year before and this was how it was presented to the team. The first go-live attempt was possible about 2 months after these famous phrases, after a lot of tweaking and testing on the initial solution.

And this was just one room.

In such a complex data transfer project, issues keep popping up at every step, sometimes baffling entire teams. “The funniest thing was when we migrated a database and ran a simple smoke test to validate that everything is working. The test failed repeatedly so we spent the whole day trying to fix the issue, only to roll back in the end. The second day we realized that we were simply testing the wrong way. We were probably a bit too nervous.”

Every castle needs its own monster and this was no exception. As one of the key issues of the project was to lower costs, the budget was the monster that needed slaying and slaying. “Traditionally, software engineers didn’t have to think about costs too much”. Having his own very healthy financial mindset for expenses at home, Sebastian Gavril explains why he took care to chip away costs at every opportunity. “With the rise of cloud providers, engineers are using cloud services more and more and taking most of the decisions in regards to the configuration of those resources. For example, the biggest databases you can order at the click of a button in AWS are ~80 times more expensive than what we chose for one of our medium workloads.”

For example, by keeping an eye on resource metrics for a database, the Levi9 team was able to cut costs by about 75% from initial estimates.

Some rooms of the castle simply got left behind and it was for the better.

One of the biggest surprises of the Levi9 team was when it realized it could take some smart decisions. The goal was to stop using our data center, not necessarily to migrate everything.

“We found out that the largest database, a 15 TB behemoth, was actually what we call “highly available”. That means, there are 2 databases instead of 1. The second one is there only if the first one crashes for various reasons. But this was not a business-critical database, so we did not need to have the db up and running 24/7. So we decided we can just drop one of the databases and rely on backups to restore the primary database in case of failure.”

It was only because of the Levi9 team’s culture of focusing on one common goal that enabled it to know the business and the client so well, that they were able to take such surprisingly easy, but important steps.

And then there was the huge data latency problem.

Levi9 practically moved the databases from the premises in Zwolle (Netherlands) to Dublin (Ireland). Some apps were left in Zwolle (Netherlands), to be migrated later, but they were calling on the database in Dublin.

“Data transfer is pretty fast nowadays, so normally this extra distance is not noticed. But due to the way one of the applications was written, the data was going to and from the database much more than needed. It became unusable, as some common operations were now taking tens of seconds to complete”

Because of the architecture of this application, it was like going to the castle for a lamp, then for the bed, then for the cover. The team was on the brink of giving up because it was not able to change the application. In the end, they managed to tweak the infrastructure so that it would allow for much more traffic than usual. It was like sending 3 people to the castle each for one thing only – not ideal, but it worked.

For a while, you will live between castles. You will still have the old one, but also the new one, perfectly in sync. And then?

And then you just need to make up your mind and throw a welcome party!

In data migration, the welcome party is extremely tricky. The inhabitants must not notice any difference, almost like they would be teleported. The service cannot be down for more than 15 minutes. The data about each Tinka client must be precisely the same. And, God forbid, DO NOT, under any circumstances, mess up their bank account. Good luck!

When it was time for the last final move, Sebastian Gavril did what he always does. He planned for it. Sebastian Gavril and his team had a step-by-step, second-by-second to-do list.

How detailed should this plan be? “Just to give you an idea”, says Sebastian Gavril, “we prepared so thoroughly that one of the go-live plans had 66 tasks, each representing an action.”

There are hundreds of small decisions that need to be taken before the key moment: for example at what time do we go live? And there are numerous connections that should be made between databases and services, all with the purpose of making a seamless transition, with as little downtime as possible. Maybe most importantly, what to do not if, rather when things go wrong.

Sebastian Gavril and his team split the very detailed action plan into three: before go-live (checking everything is in order at the new site), during go-live (making sure that each team member knows what to do at each moment) and after go-live (monitoring the new setup and being ready to take action if needed).

It all ends well, as it should.

For a result-driven, impact-delivering, change-making company like Levi9, failure was not an option at any point. No data was lost. Customers of Tinka might have noticed a two to a three-minute website glitch, but nothing more.

“Having managed to deliver this feat was a great self-accomplishment. It’s also the kind of project that challenges and empowers a senior engineer to grow, because it involves many aspects that are not part of the day-to-day life: coding, database administration, database development, cloud, networking, DevOps, architecture, third-party management, finance. A very good opportunity to develop as a well-versed engineer.”

To move critical systems, to make everything run better, to significantly increase budget savings, all the while protecting the client from any negative consequence — this could all be summarized in what Levi9 wants to achieve: impact.

The CI/CD process of an application is crucial to its success and having an easy way to maintain said process can lead to an even healthier lifecycle for the application. Enter infrastructure as code: a reliable method of managing and provisioning pretty much anything you would need in a CI/CD pipeline to get it going, through the use of templates and definitions. Ever needed to keep the configurations of an AWS pipeline somewhere so you don’t need to remember the clicks from the Management Console by heart? Or maybe you wanted to give a working example to a colleague, so they can build their own pipeline. These problems and many more can be solved through infrastructure as code and CloudFormation if we’re talking AWS.

In the following lines, we’ll go through everything you need to create your own pipeline, connect it with other pipelines and maintain them only by running a not that complicated bash script. And by the end, you’ll probably come to realize how awesome infrastructure as code is (no need to thank us).

Building a pipeline in AWS’s CodePipeline is pretty simple and conceptually similar to other CI\CD tools out there, but it’s also quite verbose, requiring a considerable amount of code, especially if you want to build your pipeline in an infrastructure-as-code approach.

From a high level there are 5 main types of resources we need in order for us to put together an AWS Codepipeline:

1. An S3 bucket resource — where our code artifact will be stored
2. A CodePipeline resource— this will model what steps and actions our CodePipeline will include and execute
3. IAM Roles — one role that CodePipeline will assume during its execution, in order to create/update/deploy the resources in our codebase. A second role is used by a CodeCommit webhook(see #5).
4. CodeBuild Project(s) — are used by the CodePipeline resource to execute the actual commands we want in our pipeline.
5. An Event Rule — an AWS Event Rule that will act as a webhook triggering the pipeline on each master branch change(this is only required when working with CodeCommit. If you use Github or other supported repo providers there are build-in webhooks)

Now we’ll go over each of the resource types and then we’ll put it all together in a complete CodePipeline.yml definition for a serverless application built on AWS Lambdas using the Serverless Framework(but not limited to these).

S3 bucket definition
You can consider this the code artifact store for our pipeline. This will be referenced and used later on by the pipeline itself to first upload the artifact and then use it to deploy the resources.

The definition is pretty straightforward:

This will be the bulk of our pipeline definition code that will define what are the stages of our pipeline.
The top-level properties of CodePipeline resource definitions are:
1. ArtifactStore — this is where we’ll reference the S3 bucket that we’ve created earlier. We will do this by using AWS’s !Ref intrinsic function
2. RoleArn — this is where we’ll reference the Role that the CodePipeline will assume during its run. We’ll do this by using the !GetAtt intrinsic function
3. Stages — a list of actions that our pipeline will do

A high-level view over our CodePipeline definition will look like this:

Notice that, for the sake of simplicity, we’ve left out the definition of the actual stages, which we’ll cover later on. But as you can see, in our case, for a serverless application, the stages of our pipeline will be:

Source — retrieving the sources from a supported git repo(CodeCommit, GitHub, Bitbucket) or an S3 bucket(this is useful if you’re not working with one of the supported git repos — but this implies having to upload your code to the bucket by your own means). We’ll stick to using a git repo since this is the most common scenario.
Staging — reference to the Deploy-to-Staging CodeBuild project which will contain the actual deploy commands.
Promote to Production manual approval gate — this will prevent an automatic deploy to Production each time the pipeline runs.
Production Deploy to Production — reference to the Deploy-to-Production CodeBuild project which will contain the actual deploy commands.

CodeBuild is the equivalent of build runners from other CI\CD tools. In our case, we’ll use their ability to run CLI commands in order to deploy our application. The definition of a CodeBuild project that deploys our serverless + node.js app to the staging environment/stack looks like this:

The most important part of a CodeBuild project is its buildspec.yml file which defines the actual CLI commands that the project will execute. You can see it being referenced in the Source property. The buildspec.yml file(located in the root of the project) looks like this:

Putting all the pieces together here’s what the definition of the stages inside our CodePipeline definition will look like:

As mentioned earlier, we’ll create 2 IAM roles. One to be used by CodePipeline itself and one used by the CodeCommit webhook that we’ll create in a future step.

An IAM role is just a collection of policies or access rights that a certain resource is allowed to have. In our case, this is the place where we define the access limits of the CodePipeline. This is important because, according to the AWS Well-Architected framework, a resource should be limited to accessing only the resources it’s supposed to access, as per the principle of least privilege. In other words, we should avoid defining IAM policies that give unnecessary privileges, like the one below:

and instead strive to limit the access boundaries through policies that look like this:

There is no silver bullet when creating a CodePipeline IAM role because the policies(access rights) of the role will differ based on the actual resources that you use in your project. There are some policies that will be required regardless of your setup as you can see below:

You can use the above definition as a starting point. Run your pipeline and add more permissions to the role based on the error messages received from AWS. Does your setup contain lamdas? Then probably you should add some lambda permissions. Dynamo db? Then add the necessary dynamo db permission. It’s a bit of a tedious process but it will add to the security of your environment.

At this point, our CodePipeline is ready. The only thing missing is its ability to run on every change of the master branch. For this, we need just two things: an Event Rule and a Role. The “webhook” we’re creating is actually an Event rule that listens for CloudWatch events emitted by CodeCommit and triggers our pipeline whenever there’s a change in the master branch(or any branch, for that matter). Fortunately, these are less verbose and look like this:

At this point, you should have everything ready and we can deploy the pipeline. This is pretty straightforward and can be done by running the following command:

$ aws cloudformation deploy –template-file .yaml –stack-name –capabilities CAPABILITY_NAMED_IAM –region

Note: the — capabilities CAPABILITY_NAMED_IAM is just an acknowledgment that you are aware that the template file will create named IAM roles.

If you’re using The Serverless Framework, there are some changes that you have to do in the serverless.yml file. Normally, your serverless.yml file looks something like this:

Notice the profile:default property — this dictates which user/profile will be used by The Serverless Framework. The default profile is usually taken from the .aws/credentials file or environment variables. The default profile is usually an admin user with full privileges.

But once the pipeline tries to deploy the serverless framework stuff, the default profile means nothing, because there are no credentials config files in our pipeline.

So we have to make use of the cfnRole property offered by The Serverless Framework. This property accepts an IAM role ARN as value and uses it when deploying the AWS resources. So we just have to put the ARN of the role we’ve created earlier in the cfnRole property, remove the profile property and we should be set. (This means, that we’ll need to deploy our pipeline template in order to create the role, find its ARN, and update the serverless.yml file)

See below the cnfRole property working along with the profile property by using the serverless-plugin-ifelse. This makes it work on staging/production environments/stacks(when CodePipeline does the deployment) as well as development stacks(when you want to deploy your stack from your development machine).

We have seen how we can declare various resources and tie them together into a fully functional, independent pipeline. But what if we want to build a much larger template by combining multiple smaller ones? Or do we want to group similar resources, such as roles, lambda configurations, or policies? AWS’s answer to these questions is simple: Nested Stacks.

Nested Stacks are stack resources part of bigger stacks. They are treated as any other AWS resource, thus helping us avoid reaching the 200 resources limit of a stack. Also, they offer functionalities such as input and output parameters, through which the stacks communicate between themselves. In addition, the use of Nested Stacks is considered a best practice, as it facilitates reusability of common template patterns and scales well with larger architectures (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-nested-stacks.html).

So how can we make use of these fancy, well-regarded, Nested Stacks? Simple! Treat each of the 3 pipelines as an independent nested stack. Since they don’t contain that many resources (now or shortly, for that matter), segregating them by type doesn’t offer that big of an advantage and thus, the nested stacks won’t even require the use of input and output parameters, since they contain everything they need inside their definition. The 3 nested stacks would reside under a parent one, used as a single point of access whenever changes are made to any of them.

What needs to be done, then? There are 3 steps we need to follow to build our template with the use of nested stacks:

This file will reference all the other templates using the physical path:

This file is a simple collection of AWS::CloudFormation::Stack resources, referencing each template through its physical path. Here, you can also define the input/output parameters mentioned earlier, which are values passed between stacks when they need to communicate, but for us, it’s not the case.

Once we’ve finished configuring our Nested Stacks setup, we can start combining them. Currently, Amazon only supports combining templates into larger ones using S3 buckets. The local files are uploaded to an existing bucket (passed as a parameter) and a new file is generated, this time containing not the physical path, but the S3 location URL

Running this command:

$ aws cloudformation package –template-file /root-template-file.yml –s3-bucket bucket-name –output-template-file my-packaged-template.yml

• — template-file — the root template file, containing the physical paths
• — output-template — the name and location of the newly generated template, containing the S3 paths
• -s3-bucket — the name of the bucket used for packaging the files
  will result in something like this:

Note: The S3 bucket used for holding these files needs to already be created, before the execution of the “package” command. Also, one needs to be careful not to include the definition of the deployment bucket inside any of the templates, since this would lead to a circular reference.

Once the output template is generated, we can safely deploy it. CloudFormation will look for the specified files in the S3 bucket and create/update the root stack and, implicitly, the nested stacks. If the stacks already exist, they are evaluated based on changesets and if any differences are found, CloudFormation updates only the ones that were modified.

The deploy command goes like this:

$ aws cloudformation deploy –template-file /my-packaged-template.yml –stack-name my-stack-name –capabilities CAPABILITY_NAMED_IAM –region my-region

And that’s about it. You should now have 3 different pipelines created by your template. Not the smoothest process, but pretty straightforward, nonetheless. A possible solution could be automating this whole endeavor through a script. In the following section, we will see exactly how we can achieve this.

As we’ve seen earlier, there are a couple of steps that we need to do to get our nested stack template packaged and deployed. Unfortunately, we have to go through all of the above processes each time we modify our pipeline.

Besides being cumbersome, doing all of these steps manually is not recommended as it is prone to errors. After all, you’re creating an automated CI/CD pipeline to reduce the amount of work you have to do, not add to it. If you’ve reached this point and you’re asking yourself “do I have to do all of that every time I want to deploy my pipeline?”, then don’t worry because the answer is no. But how can we avoid this hustle and automate the entire process? The solution? Bash scripts to the rescue!

Using a bash script, we can achieve the same result as manually deploying the pipeline(s), without giving ourselves a headache. Take a look below at an example of a simple script that does everything we need:

While this would probably work just fine (assuming the bucket exists and the template is valid), it’s a good idea to follow certain conventions regardless of the programming language you use. Let’s take a look at how we can improve our script a bit:

The above bash script does a couple of things:

Make sure the bucket exists
Because of the way nested stacks work, we need to have an S3 deployment bucket where our templates will be stored to be later used by the root stack in the deployment process. Therefore the first thing we need to do is to ensure that the bucket exists. The head-bucket command (aws s3api head-bucket –bucket $bucketName ) is perfect for this because it determines whether the bucket exists and if we have permission to access it.

Validate the template
The next step is to make sure that the template that we’re going to deploy is valid. To do this we can use the validate-templatecommand (aws cloudformation validate-template — template-body $pathToTemplate) which, if the template is not valid, will return a error message detailing what is wrong with it. Once we confirmed that the template is good, we can move forward and deploy it.

Package the template
The aws cloudformation package — template-file $pathToRootTemplate — output-template $pathToOutputTemplate — s3-bucket $bucketName command returns a copy of the root template, replacing the references to the local template files with the S3 location where the command uploaded the artifacts. So basically, it sets us up for the next step, the actual deployment.

Deploy the pipeline(s)
After all this setup, we can finally deploy our pipeline(s). We do this with the deploycommand (aws cloudformation deploy — template-file $pathToOutputTemplate — stack-name $stackName — capabilities CAPABILITY_NAMED_IAM — region $region), which uses the template that was generated by the package command in the previous step to create our (pipeline) resources. If this step succeeds, the pipeline(s) resources will be created in the specified stack.

And that’s it. We now have a script that does all the heavy lifting for us. All that’s left is to add the script to your package.json’s scripts section and you’re all set.

Quite a ride, wasn’t it? We’ve seen how to write the definition of an AWS pipeline and all its components, we’ve rigged a bunch of them together using CloudFormation and Nested Stacks and finally, we’ve automated the whole process through the use of a bash script. Hopefully, all of this came in handy and helped you avoid too many of those pesky configuration item changes on AWS when building your pipeline (guess what gave us an unusually hefty bill at the end of the month).

If you have any feedback for the article or the code presented in it, please send us an email. Every thought and idea is appreciated.

Thanks for reading and happy coding,

Andrei Arhip
Tudor Ioneasa
Andrei Diaconu, .NET Developer @Levi9