Cloud Levi9 article

Azure Skies and Smooth Cloud Sailing

 

130 years ago, when our client first started, a cloud may have been a bad omen. It could have meant the impending approach of a storm preventing its ships from reaching their destination. Today? The cloud is what helps our clients make the best decisions, based on hard, consistent, relevant data. One of these vessels shipping BI services through the cloud is called Levi9, with a team helmed by Carmen Girigan, Senior Data Engineer, and Eliza Enache, Medior Data Engineer.


From Data to Information

If a business founded in 1890 is still thriving today, it must mean that it has adapted over and over again. More recently, this meant adopting a cloud-based data solution with the help of Levi9 and other cloud providers. Vroon, an international company with a vast tradition in naval transport, was looking for ways to optimize their data solution, in order to make better decisions. We picked up a legacy on-premises solution and transformed it into a personalized, highly-scalable Single Source of Truth in the cloud.

The main challenge was picking up a complex solution using external data sources, requiring maintenance and development on multiple modules, and offering limited aggregated results. To learn how the business was doing, Vroon business users had to look into multiple sources.

However, data is mostly useless unless it becomes information. But what does this mean?

“You might look at Data as the input that your brain perceives through the senses”, says Eliza Enache, one of the Levi9 “sailors”. “The brain then creates connections between these data points by applying certain logical processes (filtering, algorithms, etc.). The connections between data points (and connections of connections) might be called Information.

Raw data is useful only when it is organized and interpreted in such a way that it helps in decision making, becoming information” completes Carmen Girigan, also a member of the Levi9 team.

After working with Levi9, Vroon users interrogate one service and get the answers they need.


From SQL Server to Azure cloud solutions

The Vroon business had been getting information using on-site Microsoft technologies, relying heavily on SQL Servers. Data was coming in from various siloed sources such as accounting, procurement, invoicing, customers, vessel activities, incident reporting, and many others.

Data were extracted, transformed, and loaded into the warehouse, via Microsoft SQL Server Integration Services, as well as custom applications and SQL Server stored procedures. On top of the data marts, there were 5 analytical databases with 11 multidimensional OLAP cubes, developed with Microsoft SQL Server Analysis Services. These cubes would then be accessed via the reporting tool, for reports, dashboards, and analysis.

The cloud conversion started in 2020. The new solution was based on Microsoft Azure technologies, in order to streamline extract-load-transform (ELT) processes and maintenance, by using a gradual approach that would allow evaluation and constant adjustment to new situations, as well as high availability for the users’ new requirements.

Now, data is extracted from its sources via Azure Data Factory and loaded into an Azure Data Lake container. It then goes through a transformation process using Azure Databricks or Azure Synapse Analytics. Afterward, it is loaded into an Azure Synapse Analytics data warehouse, based on which a tabular model is developed with Visual Studio and later deployed to Azure Analysis Services. Power BI is used for data presentation and user self-service report creation.

The result was the transformation into a modern, cloud-based, and highly-scalable solution, that provides information in a unified way: a Single Source of Truth for multiple business areas.

Having a single source of truth is essential for enabling good business decisions. “Everybody sees the same figures, at the same time, and it’s more reliable” Carmen explains. “Long story short, you avoid conflicting ‘versions of truth’ when different users analyze data exported at different points in time”, adds Eliza.


Cloud sailors tip: Clean your data like a ship’s floor

Whenever thinking of a ship, one image inevitably comes to mind: sailors cleaning the deck, almost to the point of obsession. In the 18th century, this was done to keep the crew healthy.

Today’s Levi9 data engineers have a similar attitude towards data: it should be maintained clean, to avoid the risk of infecting the budget, reports, and any meaningful analysis.

In the cloud, resources are much easier to maintain and scale — but we do have to pay attention to costs. There is no point in having duplicated or unutilized data, or transformations, in the cloud”, explains Carmen. Eliza adds that there is much more than that. “During the testing phase, we noticed that redundant or obsolete data led to improper analysis and estimation, and diversion from the ‘main track’ — which in turn led to an increase in time and costs, while also blocking further developments. Some of these redundancies made us go back through the whole logic.”

Those who don’t empathize with nautical experience might compare data cleansing with their wardrobes: “Just think of that closet full of clothes you don’t like to wear, things you’ve kept for years only cluttering your space and making for a hefty baggage” says Eliza.


An ongoing adventure

Part of Vroon’s legacy system is still running and is in the process of being transformed. Eliza explains why sometimes this is a better solution: “We soon realized that converting and rewriting the whole logic, from old technologies into Azure, would take a lot of time – thus stalling any further development initiative. As such, we devised a temporary workaround: part of the data is directly brought to and processed in Azure, while another part is brought from the on-premises data warehouse, and further integrated into the unified model.”

Cloud Azure Levi9 Articles

But we won’t do everything at once. First, we’ll do a pilot project”, adds Carmen. Just like sending a sentinel ship before moving the entire fleet, a pilot provides insight into what the job entails.

Cloud Azure Levi9 Articles

Fair winds and following seas

All in all, the project continues to be a learning opportunity for the team. All the Levi9 Vroon sailors have at least one Azure certification, as the department and company strategy aligned with the work. Carmen Girigan and Eliza Enache stress that they owe much of their learning to the other half of the initial engineering team that had established the foundations on which they continued to build in the cloud.

With the new pilot project, the adventure continues. May the team have fair winds and following seas!

We’re supporting your race at your own pace. Choose yours!


AWS Levi9 Articles

Rebuild this castle - a data migration story

Imagine yourself in a run-down house. Make it a big one. Huge. Some walls are still standing, but barely. Some rooms are just a pile of bricks lying in the middle of the floor. There are secret passageways connecting random chambers, almost like a labyrinth. Did you make the house gigantic in your mind? Go on, make it even bigger. A quirky, dysfunctional, enormous castle.

Here is your challenge: you need to rebuild the castle on another land. In its new version, the building should regain its splendor, while guests and inhabitants alike should be able to find their way seamlessly. Oh, and if you fail, even slightly, even by a fraction, all the inhabitants of the palace lose their livelihood.

Proceed.

This is the magnitude of the challenge that software architect Sebastian Gavril found himself in, when he was approached to migrate a fintech’s core databases to the cloud.

Tinka, a Dutch fintech company, already had some of its services in the cloud, such as microservices, authentication systems, message brokers, CI/CD, and monitoring tools. But the core services, related to the actual financial processes, remained on-premise, part of the so-called legacy system: the virtual machines, databases, the Enterprise Service Bus, and workflows.

The job entailed moving from on-premise into the cloud 30+Tb of data, split in over 30 Oracle database instances across multiple Exadata machines — mini data centers that can be rented out — and 7 environments.

In other words, the core services were still in the “old castle” and needed to be moved. All those discarded bricks, secret passageways — they were actually scattered databases on old servers, connected among each other in a complex network, being used by various applications, microservices, or monitoring tools.

So how does this story unfold?


The heroes

This job needed an architect. A software architect. “Normally, an architect first needs to understand his customers, their needs and desires, constraints and opportunities. Then he translates that into a formal language that can be used by engineers to construct the building his customers desire. And during the construction phase, the architect is always present to make sure the plan is followed and change requests from clients are included with minimal impact on the integrity of the building.”

This is what Sebastian did. He needed to have an ensemble view of the challenge, decide what’s needed, and get the team together. They were: Simona Gavrilescu (Delivery Manager), Ancuta Costan (Test Lead), and Monica Coros (DevOps Engineer), as well as employees of Tinka.

At Levi9, the team is defined together with the customer, with the aim of enhancing collaboration and working together as one team.


The castle

Moving core databases is an extremely complex project. When you move databases, you don’t just move the data, but you need to make sure that all the critical systems will function correctly.

Plus, the old castle — the legacy systems — were responsible for a large part of Tinka’s income. This is true for most companies.

While looking to achieve a positive impact for the clients, one of Levi9’s goals of this project was to have no negative effect on the Tinka customers. Customers were not supposed to notice anything. However, if things did go wrong, the company could have lost or corrupted customer data.


The new kingdom

So where was this new land where the castle was being built? The cloud.

Until recently, “the cloud” actually meant just a couple of key players, such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform. Nowadays, any large, established company starts to develop its own cloud: Oracle Cloud, IBM Cloud, Digital Ocean, Heroku, Alibaba. To attract customers, many offers features such as niche-specific services, better integration between cloud and legacy systems, and migration tools.

The chosen solution was the RDS – Relational Database Services, the database service offered by AWS, mainly because part of Tinka’s infrastructure was already in AWS. After migration, AWS would be the one keeping up with maintenance and updates and many more as part of the package.


The challenge

“It was imperative to understand the complexity, which I saw on two levels”, says Sebastian.

The first level of complexity is that of the database. In this case, the team used a tool provided by the AWS Data Migration Service (DMS) of AWS called the Schema conversion tool. “When you move a database from one place to another, this tool generates a scheme that shows the objects, triggers, procedures, and packages. It also shows which ones can be converted automatically and where manual changes need to be made, through simple, medium, or complex actions. This helps us understand what database-specific effort we need” explains software architect Sebastian Gavril.

The second level of complexity is that of the “landscape”. That means everything that calls on the database. The landscape was very rich. From old, legacy servers – that nobody really understood, to modern microservices, with everything in between: ESB, custom jobs, etc.

It helps us understand where the problems are. What we need to change: connection strings, drivers, maybe even code that used specific functionality in Oracle Enterprise Edition (an edition we wanted to move away from).


The map

So what does moving the castle entail? Well, when planned systematically, things look pretty straightforward: you need to move the bricks and you need to redesign the rooms. In data transfer language this means that you need to take into consideration two aspects: the applications and the databases.

On the applications side, there are some conversions to be made: for example update libraries, change versions, rewrite some code, and make sure the configurations are in order.

The databases also need to be converted, either by using tools such as the schema conversion tool or manually. For some, you can find a procedure. For others, they need to be built from the ground up.

For migrating the actual data, just like renting a moving truck, AWS has a Database Migration Service offer. You can rent a machine, point to the “departure” address, set an “arrival” address, and up you go. It can take minutes, hours, days. Or weeks, in one of our cases.

AWS Cloud Levi9 Articles

It all comes crumbling down - Part One

Every project starts with “it’s not much”. “It’s practically done”. Sebastian Gavril knows to cringe at this one. The first migration project was started 1 year before and this was how it was presented to the team. The first go-live attempt was possible about 2 months after these famous phrases, after a lot of tweaking and testing on the initial solution.

And this was just one room.

In such a complex data transfer project, issues keep popping up at every step, sometimes baffling entire teams. “The funniest thing was when we migrated a database and ran a simple smoke test to validate that everything is working. The test failed repeatedly so we spent the whole day trying to fix the issue, only to roll back in the end. The second day we realized that we were simply testing the wrong way. We were probably a bit too nervous.”


Slaying the budget monster

Every castle needs its own monster and this was no exception. As one of the key issues of the project was to lower costs, the budget was the monster that needed slaying and slaying. “Traditionally, software engineers didn’t have to think about costs too much”. Having his own very healthy financial mindset for expenses at home, Sebastian Gavril explains why he took care to chip away costs at every opportunity. “With the rise of cloud providers, engineers are using cloud services more and more and taking most of the decisions in regards to the configuration of those resources. For example, the biggest databases you can order at the click of a button in AWS are ~80 times more expensive than what we chose for one of our medium workloads.”

For example, by keeping an eye on resource metrics for a database, the Levi9 team was able to cut costs by about 75% from initial estimates.


Time to say good bye

Some rooms of the castle simply got left behind and it was for the better.

One of the biggest surprises of the Levi9 team was when it realized it could take some smart decisions. The goal was to stop using our data center, not necessarily to migrate everything.

“We found out that the largest database, a 15 TB behemoth, was actually what we call “highly available”. That means, there are 2 databases instead of 1. The second one is there only if the first one crashes for various reasons. But this was not a business-critical database, so we did not need to have the db up and running 24/7. So we decided we can just drop one of the databases and rely on backups to restore the primary database in case of failure.”

It was only because of the Levi9 team’s culture of focusing on one common goal that enabled it to know the business and the client so well, that they were able to take such surprisingly easy, but important steps.

AWS Cloud Levi9 Articles

It all comes crumbling down - Part Two

And then there was the huge data latency problem.

Levi9 practically moved the databases from the premises in Zwolle (Netherlands) to Dublin (Ireland). Some apps were left in Zwolle (Netherlands), to be migrated later, but they were calling on the database in Dublin.

“Data transfer is pretty fast nowadays, so normally this extra distance is not noticed. But due to the way one of the applications was written, the data was going to and from the database much more than needed. It became unusable, as some common operations were now taking tens of seconds to complete”

Because of the architecture of this application, it was like going to the castle for a lamp, then for the bed, then for the cover. The team was on the brink of giving up because it was not able to change the application. In the end, they managed to tweak the infrastructure so that it would allow for much more traffic than usual. It was like sending 3 people to the castle each for one thing only – not ideal, but it worked.


The welcome party

For a while, you will live between castles. You will still have the old one, but also the new one, perfectly in sync. And then?

And then you just need to make up your mind and throw a welcome party!

In data migration, the welcome party is extremely tricky. The inhabitants must not notice any difference, almost like they would be teleported. The service cannot be down for more than 15 minutes. The data about each Tinka client must be precisely the same. And, God forbid, DO NOT, under any circumstances, mess up their bank account. Good luck!

When it was time for the last final move, Sebastian Gavril did what he always does. He planned for it. Sebastian Gavril and his team had a step-by-step, second-by-second to-do list.

How detailed should this plan be? “Just to give you an idea”, says Sebastian Gavril, “we prepared so thoroughly that one of the go-live plans had 66 tasks, each representing an action.”

There are hundreds of small decisions that need to be taken before the key moment: for example at what time do we go live? And there are numerous connections that should be made between databases and services, all with the purpose of making a seamless transition, with as little downtime as possible. Maybe most importantly, what to do not if, rather when things go wrong.

Sebastian Gavril and his team split the very detailed action plan into three: before go-live (checking everything is in order at the new site), during go-live (making sure that each team member knows what to do at each moment) and after go-live (monitoring the new setup and being ready to take action if needed).

AWS Cloud Levi9 Articles

The happy ever after

It all ends well, as it should.

For a result-driven, impact-delivering, change-making company like Levi9, failure was not an option at any point. No data was lost. Customers of Tinka might have noticed a two to a three-minute website glitch, but nothing more.

“Having managed to deliver this feat was a great self-accomplishment. It’s also the kind of project that challenges and empowers a senior engineer to grow, because it involves many aspects that are not part of the day-to-day life: coding, database administration, database development, cloud, networking, DevOps, architecture, third-party management, finance. A very good opportunity to develop as a well-versed engineer.”

To move critical systems, to make everything run better, to significantly increase budget savings, all the while protecting the client from any negative consequence — this could all be summarized in what Levi9 wants to achieve: impact.

We’re supporting your race at your own pace. Choose yours!


10 ways Java is getting better

You might think of Java as outdated. It is, indeed, 27 years old, which is the equivalent of one century old in “IT years”. Its younger, fresher, hipper siblings — such as Scala, Kotlin, Closure, or Groovy — might be more dynamic and easier to use. But don’t discard Java just yet!

Andrei Micu, Senior Scala Developer at Levi9, has noticed that Java is evolving and using its younger siblings for inspiration and improvement. Andrei talks here about two projects that intend to evolve the JDK, with their associated JEPs (JDK Enhancement Proposals): Project Amber and Project Valhalla.

Let’s see some instances where Java is a bit behind its siblings and how syntax sugar and standardization simplifies and makes the code more efficient.


1. Using records like in Scala or Kotlin

POJOS (Plain Old Java Objects) make data encapsulation a bit difficult. They needed a lot of writing but other languages such as Scala and Kotlin came up with features that standardized and simplified the writing. The same output could be achieved with a case class for Scala or data class for Kotlin. And this is what Project Amber tries to change.

Project Amber started with JDK15 and did some syntax sugar by adding the concept of records that brought in the same feature from Scala or Kotlin.


2. Better Pattern Matching

Another interesting change brought on by Project Amber is pattern matching. Old Java is not very flexible with switches, but Scala is way ahead. For example, in the second Scala example below — you can switch in the same object for lists, for components of lists and you can extract from them.


3. Pattern Matching for instanceof

Levi9 articles Java

The concept of pattern matching for instanceof was added in the first release of JDK14. Previously, when you checked if an object was an instance of a class, an extra line was necessary for the class to be cast to its checked type. But since JDK14 you can simply write the name of the variable after the class and you don’t need that extra line. This is a small, neat improvement.


4. Pattern Matching for classes in switch

Levi9 articles Java

JDK17 brought on pattern matching in a switch. If you want to check an object for its class type, now it’s possible in switch statements too.


5. Increased Expressiveness

Levi9 articles Java

JDK18 came along with increased expressiveness. Not only can you now check classes in switch statements, but you can also check for particular values and classes in the same switch. Guards and nulls can also be added.


6. Introducing Record Pattern (Preview)

Levi9 articles Java

In the future, we will also have record patterns. Previously, if we got a record at this point you had to write several lines of code to access its members, but now you can do an instanceof point with the name of the internal fields and it’s going to extract them for convenient use, afterwards.

In switches you can do the same for cases
. This sort of expressiveness is going to deconstruct that point, which is something pretty new in Java, so that’s why it probably takes more time to cook.


7. Classes for the Basic Primitives (Preview)

We are now in the territory of primitive types. This is the focus of Project Valhalla. In Java, primitive types are enough to model almost anything. But anything that’s not on the primitive type list is derived from Object, and this is quite an issue. Scala and Kotlin have a different approach to this. Both languages use just one class and the compiler does the improvements for you.

However, Java does not have a class for primitive types, making it difficult to work with them seamlessly.

But this may change and primitive types might get a class. This class will be marked with a `Q` prefix in the JVM internals to signal it’s a primitive type.


8. Declaring Objects that don’t have identity (Preview)

Java wants to offer the possibility of making objects as primitive types. The first step is to give the possibility of declaring objects that don’t have identity.

But what does “having identity” mean? Long story short, when an object has identity, two instances of the same value are different. This is not the case for primitive types and this is why we say that primitive types do not have identity.

Furthermore, objects that don’t have identity are stored in the stack, while objects with identity are stored in the heap.

Value objects (or classes) don’t have identity. Let’s look at how they are declared.

Levi9 articles Java

While the outline of the definition is still work in progress, there are some definition rules that have been already established. One is that the class should be implicitly final. The fields of the class are implicitly final and no constructor can call super. Also, value objects can have reference cycles, which means they can have references to another object of the same type.

What do we get in return? The objects are put on the stack and the equality will hold between two references to the same value..


9. Primitive Classes (Preview)

Levi9 articles Java

A primitive class is like a value class with the additional restriction that no field can be of the declaring class, so we don’t have cycles like we have for value objects.

What do we get in return? We get almost the same performance as primitive-types. They do not allow nulls. Some might say this is not a benefit, but those who had null-pointer exceptions might think otherwise.


10. Reified Generics (maybe)

While not yet official, Project Valhalla does drop a hint about tackling reified generics, in which they may also take a hint from Scala and Kotlin.

What is the issue in Java right now?
Well, when generic type objects are passed around, the information about the inner types is lost at runtime. This is called `type erasure` and it is a drawback of the JVM.

There is a workaround in Java, that requires you to send the class as a separate parameter. Definitely not elegant! But Kotlin and Scala have a better way of dealing with this.

Kotlin deals with this with a neat feature called “reified generics”. If you declare the function inline — meaning that the function content is copied at the call site — then you can have access to the inner types of the generics. When you write reified, it adds in the back an extra parameter which is the class of the generic type parameter. This way, you don’t have to write extra parameters.

Scala works around this in a different way. It makes use of implicit parameters to leverage the compiler’s ability to pass the class information.

Levi9 articles Java

What do you think?

Do these Java improvements seem significant to you? Andrei Micu stresses that standardization is not easy. Java’s little siblings might be a bit lighter on standardization, which allows them to develop features faster. Java, however, is so widely used, that keeping backward and forward compatibility is crucial. Sometimes, this means holding back on innovation, in order to keep all of its users happy.

Let’s keep in mind Java is running a marathon, says Andrei Micu. Its younger sibling languages, with more spring in their step, will also benefit from JVM improvements and new features. It’s a win-win.

We’re supporting your race at your own pace. Choose yours!