We have migrated Apache Kafka to AWS for one of the world’s leading fashion companies, as part of a large migration of the customer’s workloads to AWS. We are sharing our experience gathered recently on one of the projects, and we think you may find it interesting if you face similar challenges. Moreover, if you recognize some steps, this article is for you.

Over the years, we have operated and successfully migrated multiple Kafka workloads to AWS; still, each migration was challenging and requested an investigation and thorough thinking ahead.

Everyone who has hands-on experience with systems that use Kafka know how crucial it is for success and what role Kafka plays in capturing large volumes of streaming data.

The cloud and AWS opened new horizons regarding running these workloads. Cloud evolution brought us a wide variety of services with various levels of abstraction, suited for different use cases. The migration of existing systems to AWS has advantages, when measured against all pillars of a Well-Architected Framework, but it still needs to be well-planned to match the use case. Poorly made decisions can lead to headaches in the future.

Legacy setup

We have been challenged to migrate the Confluent Platform to AWS as a part of the client’s cloud migration initiative.

The system built on top of Mesosphere DC/OS reached end-of-life for underlying virtualized hardware, so the strategic decision was made to move to AWS. The container platform successfully ran for many years and was built with tens of nodes, hosting multiple environments. The application landscape was built using a microservice architecture with Confluent Kafka as a central backbone of the entire system; therefore, it was the most critical part.

Kafka Connect source connectors played a significant role in the system and have been used to import multiple data streams from various data sources like SAP and production databases. Data transformation is transformed with available tools, some using the Connect SMT functionality, and some fed into KSQL for more complex transformations. After preparing and processing the data, the resulting stream was exported to the NoSQL database using Kafka Connect.

This was the old setup:

• Development and production Kafka clusters – Confluent 5.3
• Zookeeper
• Kafka Connect
• KSQL
• Schema Registry
• REST proxy
• Control Center

Implementation

We started by examining the available options that we had and map those to the existing setup.

For Kafka itself, the easy decision was to use Amazon Managed Streaming for Apache Kafka (MSK), which is basically a managed Kafka cluster. It is managed, easy to set up, and you can leverage all benefits of using it.

For moving the rest, we relied on the ECS-Fargate cluster and run Connect, Schema Registry, KSQL, and REST Proxy inside managed containers.

With this approach, we managed to have a working solution that has the “look and feel” of the original one, leaving room for future improvements.

The last crucial part which we had to solve was the actual migration, and we performed it with MirrorMaker 2.0, synchronizing the data between clusters and enabling the actual migration to take place.

The first impressions after migration:

Some of the immediate advantages that we had after the move and that we want to share are:

• Elastic storage in case one of a sudden unexpected growth of multiple topics
• Flexibility to adjust the underlying instances according to additional/expected load demand
• No licenses
• Seamless “switchover” for the developers and maintainers

Lessons learned

In the lifecycle of any company there comes a time when a company outgrows its current “in-house” configuration.  Because of several reasons, AWS is the logical next step.

Cloud migration action is a complex and usually expensive undertaking, and during this process, many things can go wrong. To reduce or even mitigate the risk, always perform meticulous planning before migrating systems, even if they are not considered “core” business or important systems.

Many lessons learned are driven by the principles of the Well Architected Framework:

• START SMALL and evolve – To reduce risk, if feasible, start with a POC, migrating non-critical applications first to identify issues that may occur in advance.
• AUTOMATE EVERYTHING – Write “code” for migration and prepare CI/CDs for both the application/services and the AWS resources deployment/configuration. Minimize manual actions in the future. The system will therefore be repeatable and less error prone.
• GET TO KNOW YOUR SYSTEMS – Careful planning during the assessment phase can save you much time and minimize the risks of having to mobilize all involved, especially in the implementation of migration.
• BUSINESS IS THE DRIVING PART OF THE MIGRATION – Involve business decision-makers not just for approval of the process but preferably as a driver of planning and prioritizing what are the features that must be enabled first. Explain and motivate it not just to understand the migration process but also to provide critical feedback!
• THINK BIG, THINK FUTURE – Migration to AWS gives us a chance to provide nearly limitless scalability and service availability, tailored by the preferences and “size” of the client company, together with a premium on data security, confirmed and used by the largest companies of every industry sector.

Levi9 is an Advanced Tier AWS Consulting Partner. Levi9 has achieved the DevOps Competency and the Data & Analytics Competency. Levi9 is also part of the AWS Well Architected Program.

Get in touch with our AWS Alliance Lead Boudewijn Haas to discuss your wants and needs.

Incision, based in The Netherlands, is a health technology scale-up business that provides an online learning platform to assist surgeons in learning surgical techniques, help prepare operating rooms with the right equipment, and improve overall surgical skills and knowledge. The company has a global customer base with key markets in Belgium, the Middle East, Scandinavia, the UK, and the US.

Incision Academy helps surgeons at all learning levels and offers a standardized care approach that includes the latest techniques to help train surgeons and reduce training costs. The company provides 700 video courses for surgery, perioperative care, and anesthesia with interactive 3D anatomical models and additional media-rich information.

The company started to experience performance issues following a rise in customer numbers. Its WordPress-based and hosted website and database was struggling to scale and serve content rapidly enough for large numbers of users—with some waiting up to 30 seconds for each webpage to load. In addition, new product deployments were hard to introduce and risked causing downtime.

Because its website was hosted in Ireland only, this meant that website access outside of Europe was even slower, limiting the company’s ability to serve its global customers. Incision had also started to receive several customer complaints, so it needed to act fast to improve website access.

To resolve these issues and meet its ambitions of networking 10,000 clinics and assisting with 100 million annual operations, the company needed to be able to scale its operations.

It turned to AWS Partner Levi9 to help it migrate its website to AWS to improve performance and scalability and to use modern cloud services, such as artificial intelligence (AI) for building a smartphone app.

Reducing Technical Debt

Levi9 began working on the project in July 2019 and created a roadmap and plan for the implementation of new features. It migrated Incision’s website to AWS and built a back-office web application for user data administration, which had previously been handled in WordPress.

Innovation

Using AWS, Incision uses AI to analyze videos to assist surgeons in improving their skills, identify areas for improvement, or highlight if they have made unnecessary or incorrect actions during a procedure. This video content can also be used for training other surgeons—while anonymizing the patient. Levi9 also built a smartphone app on AWS to target surgical support staff and scrub nurses to help them prepare better for operations. This includes operating room preparation, patient positioning, and checking all required equipment is available. This helps cut the time that is normally wasted in operating rooms. Levi9 also built a data analytics platform using several AWS services to help inform product development and gain insights into which features are being used the most and least. For this it uses a combination of Amazon Kinesis to collect and stream the data, AWS Glue to prepare it for querying, and Amazon Quicksight for analysis. 

Agility & Performance, Availability, Innovation

By building on AWS, Incision can automatically scale up and to meet busy usage periods and scale down when user numbers fall. It also uses multiple AWS Regions to cache website content to ensure consistent and fast access times for all customers, wherever they are located. The company can now also track usage patterns and see how its platform is being used in hospitals to help it make product improvements. It uses Amazon CloudWatch to track user response times to ensure its website access remains efficient. Deployment is now faster and easier and can handled multiple times during a day with only a few clicks and no downtime risk—previously new deployments were much harder to achieve.

Security & Compliance

To support expansion in the US, Incision relies on the built-in compliance of AWS services. It also uses multiple AWS services to support platform security, including Amazon GuardDuty and Amazon Inspector to regularly scan its infrastructure.

Levi9 is an Advanced Tier AWS Consulting Partner. Levi9 has achieved the DevOps Competency and the Data & Analytics Competency. Levi9 is also part of the AWS Well Architected Program.

Get in touch with our AWS Alliance Lead Boudewijn Haas to discuss your want and needs.

Future Insight envisages a great future for the digitalisation of processes and activities within the public domain. Their platform helps people to make the right choices in projects involving the living environment, such as infrastructure, development and the repurposing of sites and buildings. In partnership with Levi9, vision and ambition are becoming a reality. ‘The partnership enables us to think big and also put our ideas into action,’ says CTO and co-founder Rick Klooster.

The company provides three separately developed software solutions designed to optimise collaboration and shape successful projects: Clearly.Projects for construction projects; Clearly.BIM for optimum viewing and interrogation of BIM models; and Clearly.3D-City for accessing 3D-city models (Digital Twins). All three involve software that brings together and makes available data from different sources.

‘This means that the parties involved in construction projects, works on physical infrastructure, or the life-cycle of a building can access the right up-to-date data and tools to enable smart collaboration,’ explains Klooster. ‘We have built our reputation on this. In 2022, Clearly.BIM secured us the Building Smart Award in Estonia, based on the implementation of a BIM-based planning permission process.’

With the assistance of Levi9 Technology Services, these solutions are being further developed and increasingly integrated. ‘Together, we’re building a growing ecosystem of all kinds of services,’ says the CTO. ‘This Open Urban Platform enables us to easily access specific solutions from third parties. Some of these relate to AI. As a result, we no longer need to develop everything ourselves, and other parties can serve part of their market using our platform.’

The Open Urban Platform is part of the Future Insights Clearly.Suite. It represents the combined vision and ambitions of CTO Rick Klooster and CCO Bas Hoorn. ‘It’s a dream that’s becoming a reality,’ says Hoorn, originally an engineer. For his part, Klooster has a lot of experience working with government. Future Insight has now become a genuine knowledge company with highly qualified specialists.

In anticipation of its rapid growth, Future Insight decided to expand its own development team with external knowledge and experience in 2021. Klooster: ‘The key aim is to achieve a stable and rapidly scalable platform. In view of the intended growth, we also aim to organise ourselves and our teams more effectively and professionally. That’s why we made the decision to have a partner develop our technology.’ Following a thorough search, Klooster and Hoorn came into contact with Levi9.

It has been a genuine partnership from the outset. ‘We have a similar culture and way of working,’ explains Albert Klingenberg, account manager at Levi9. ‘Future Insight are completely open with us about their vision and strategy. That’s reciprocated. This openness gives us the opportunity to keep each other on our toes and make real progress. Our people also really love working on the Future Insight platform.’

The founders are convinced that there are genuine global opportunities when it comes to sharing information in urban areas. For example, there are as yet no widely embraced worldwide standards when it comes to smart cities, buildings, areas, and infrastructures. With the help of Levi9, Future Insight is determined to and capable of playing a crucial role in this development. An important motivator in this is the desire to address this global social challenge.

It is also helped by the fact that government authorities across the world are increasingly open to the cloud. CTO Rick Klooster: ‘That will potentially result in an increasingly wider application of international standards in the future. It will then become increasingly easier even for smaller municipalities to engage and collaborate in an integrated way without any form of vendor lock-in.’

Future Insight has the wind in its sails. The company has doubled its staff numbers in the last six months alone. ‘Things are really moving, and we’re working on all kinds of things simultaneously and clearly proving remarkably successful. The people working for us based in Serbia are actually our development department. That’s where our ideas really take shape.’

Source: itexecutive.nl

For the last decade, five°degrees has been supplying leading financial institutions with a tried-and-trusted core banking product. Last year saw the launch of a completely cloud-native version that will enable customers to continue to meet ever-changing market requirements even in the long term. The partnership with Levi9 proved to be instrumental in the development of the new platform. ‘It was exactly the type of synergy we needed,’ confirm CEO Martijn Hohmann and CTO Jeffrey Severijn.

Instead of modernising the existing solution, the decision was made to redesign the new one from scratch. It is now ready for the market. ‘Unlike the old stack, the new ‘banking grade’ SaaS platform – known as °neo – is component-based,’ explains CEO Martijn Hohmann. ‘Over time, the number of building blocks will gradually increase. It will also become easier to link external services and ecosystems together.’

Five°degrees and Levi9 Technology Services have been working together for seven years, but the interaction has entered a new phase in the last four years. ‘At management and shareholder level, we’ve been considering how to get the most value out of our relationship for some time,’ says the CEO.

‘Levi9 is really determined to create value on the business side for customers, partly because that’s also interesting for their own employees.’ – Hohmann

Added value

‘Delivering value for business is the holy grail for developers,’ says CTO Jeffrey Severijn, who also has ultimate responsibility for the °neo platform. ‘Being able to offer employees a challenging working environment is important for Levi9. As far as our traditionally designed °matrix solution was concerned, our relationship was gradually entering the danger zone. Developers were just losing their enthusiasm for it.’

According to the CTO, there was also another factor at play, ‘Outsourcing partners are facing significant increases in salaries in their international branches caused by the COVID-19 pandemic. That eats into the price advantage. Companies like these now need to distinguish themselves in different ways. For example, through specific market or domain knowledge that we can make use of.’

There is added value for five°degrees if a player like Levi9, with six delivery centres in Eastern Europe, has experience with the Azure cloud, for example within a media company. ‘They can then discuss applying those competencies for other customers, which has advantages for everyone: partner, staff, and customers. It even opens up the potential of wider co-creation in the future.’

It was during a frank discussion with Levi9 about all of this that things started to move forward.

‘We had talks with various different players about modernising our existing core banking product, but we achieved very little. However, Levi9 came from a different direction and suggested adopting a totally different approach using the latest methods and techniques. That brought a fundamental change in our partnership – and one that added value for all of us.’ – Hohmann

Start-up strategy

Around 80 people – half of the total capacity – worked on °neo over a three-year period. The completely new platform features lower variable costs, offers much more flexibility and has an important role to play in the company’s global ambitions. Severijn: ‘Together, we approached it as if it was a start-up: from the initial design to a minimum viable product before going on to develop a technical MVP. All of it agile and focussing on the potential for rapid upscaling. Ultimately, it all turned out well.’

The partnership is based around the shared objectives that the two companies aim to achieve as a team. ‘It really is a joint initiative,’ say the CTO and CEO. ‘Our people sometimes struggled with that. It was no longer a case of “we ask, and they do what we say”, as you would approach it with development teams in India or Vietnam, for example. Everything is based on a relationship of equality.’

According to the management at five°degrees, this has brought out the very best in everyone, starting from a blank piece of paper. Severijn: ‘The contribution made by Levi9 was primarily technical: the Azure cloud, event-driven architecture, microservices, and so on. Our main focus was on the functional aspects. We then came up with a lot of great ideas together.’

Speeding up the process

Examples of this include work relating to Logic Apps for integrating apps, data, services, and systems using automated workflows in Azure. ‘During a demo with the key engineers from Microsoft, it turned out that we were further advanced than they were on some points,’ says Hohmann enthusiastically. ‘It was so good in fact that they even adopted some of our ideas.’

As for the partnership, the CEO is keen to stress how productive and enjoyable it is. ‘If people from Levi9 have ideas about how things can be improved, we’re always keen to listen. In return, we gave a presentation about the future of banking to their development team in Serbia.’

Jeffrey Severijn: ‘During the demos, members of the Levi9 team also made regular contributions. That helps create ownership and mutual understanding. Even when there are challenges, if something takes slightly more time, or if the requirements are unclear. The fact that we already knew each other allowed us to speed up the process that little bit. Ultimately, developing a new relationship always takes time – but we understand each other’s strengths and weaknesses.’

‘All in all, it was exactly the synergy we needed to make a success of the °neo project,’ concludes CEO Martijn Hohmann. ‘I’d even go so far to say that it would have been impossible without it.’

Source: itexecutive.nl

Incision has become a worldwide success with its video-based education and training for operation room personnel. Now, alongside its trusted e-learning environment, Incision has launched a new digital assistant, developed in partnership with Levi9. This tool went from idea to market in record time. ‘We started in June 2021, and four months later, it was live,’ says Raimo van der Klein, Chief Product & Technology Officer. “And a year later, we were already running pilots in American hospitals.’

The mobile app is a strategically important expansion of the Amsterdam scale-up’s existing e-learning platform. ‘Now we can offer users direct support before and during the surgery,’ says Van der Klein, who stepped into his role as CP&TO two-and-a-half years ago. ‘This new digital product is crucial for the relationship with the end user.’

Incision Academy has been around for a while, offering e-learning accredited by international medical associations to train medical professionals in operation room-related activities. Now, the new Incision Assist tool gives those professionals direct access in the operation room to all relevant information: instructions, manuals, requirements, and the personal preferences of the performing surgeons – and all this for a wide range of procedures.

Co-creation

Incision Assist was developed in record time in a process of co-creation with Levi9: from concept to worldwide market launch in under one year. ‘The challenge was that there was relatively little we could reuse from our existing e-learning environment. And on top of that, with Assist, we’ll become part of the digital infrastructure of the hospital, so part of a strongly regulated IT environment. That requires enterprise-grade software.’

Incision was founded in 2014 by Dr Theo Wiggers and a group of investors. It is now a cutting-edge collective of driven doctors, software developers, marketers, and professionals working to grow the business from its home base in Amsterdam. ‘We believe in sharing surgery-related knowledge and skills and making them available to everyone,’ says Van der Klein.

The newly developed app will be a boon to students, personnel in training, and temporary staff in particular. ‘With Incision Assist, we can help operation room staff with any process and any procedure, in a uniform way. Its advantages are better preparation, reducing risks, improving team functioning and, ultimately, better medical outcomes. In combination with Incision Academy, this gives us a rock-solid proposition.’

Increasing impact

About the co-development process with Levi9, Van der Klein says, ‘They had also been involved in the development of the existing e-learning platform. From the moment that we decided to increase our impact in the digital domain, we worked out the best approach in a very effective dialogue. The goal was to, in a relatively short amount of time, put out a product that was at least viable, and that we could then scale up to the highest quality requirements fast and efficiently.’

Incision and Levi9 as partners went through a hyper-fast learning phase in a working process in which they used feedback from customers and users to add more and more features to the app. ‘At this point, we’re going forward with the expansion and upscaling of the platform based on modular technology.’

Van der Klein calls the joint development process a transformation within a scale-up. ‘And that in an industry that’s super-complex, that’s struggling to fill jobs right now, that’s heavily regulated, and that has extremely high standards for security. Because of these challenging and above all complex dynamics, we knew we needed a technology partner that could give us the right people, quality, and working methods. Without Levi9, we’d never have been able to do this as well or as fast.’

Source: itexecutive.nl