Engineering Practice · Brennenstuhl on Security

Goodbye Big Tech: My European JAMstack Migration

Sun, 06 Jul 2025 00:00:00 +0200

I recently migrated my personal JAMstack projects from Github and Netlify to European alternatives. In this article, I’ll share which European service providers I went for and why I sleep much better now. Before we begin, though, let’s quickly review what we mean when we use the term JAMstack!

What is JAMstack? #

A JAMstack is a software architectural approach that is solely based on frontend JavaScript, APIs and Markup (typically generated by a static site generator). This blog is an excellent example of a JAMstack website, as the client-side contains all the application logic (primarily search), while the backbone is composed entirely of static files. There is no classical database or backend application – everything is static files delivered to user agents. Following this architectural design yields significant benefits:

minimal attack surface,
effortless maintainability,
blazingly fast performance, and
easy scalability.

The term JAMstack was initially coined by the US SaaS provider Netlify. They and Github Pages are two of the most prominent service providers in that ecosystem. However, there are European alternatives.

My European JAMstack #

For my European JAMstack setup that is based on the excellent static site generator Hugo, I looked for service providers that not only provide great service but also support data privacy and open-source principles, where applicable.

Codeberg: My New Home for Code #

For code hosting and deployment, I opted for Codeberg. Github, while ubiquitous, is a US-based company owned by Microsoft, and I wanted to move my repositories to a European alternative. Codeberg is a non-profit organization based in Germany dedicated to providing a free and open-source Git code hosting platform. It’s powered by Forgejo, which in turn is a hard-fork of Gitea.

Codeberg.org offers a familiar interface for anyone used to Github and Github Actions. It is incredibly reassuring to know that my code resides on a community-driven and open-source platform rather than a “Big Tech” for-profit one.

→ The source files of this blog have moved to GitHub. The update at the top of this article explains why.

Bunny CDN: Fast and European Hosting #

Replacing Netlify was a crucial step, as it used to handle the deployment and hosting of my static files. Based in Slovenia, Bunny CDN offers amazing CDN capabilities that easily outperform Netlify. It’s a super robust and comprehensible content delivery platform and an ideal solution for hosting static sites. Their global network ensures excellent performance with the added benefit of being a European company.

Bunny’s user interfaces are generally easy to use, and their prices are reasonable – I am paying single digit Euros every month for serving all my projects. Knowing my websites are hosted with a European-headquartered provider, with a clear focus on GDPR compliance, significantly contributes to my peace of mind.

As JAMstack deployment essentially means triggering a static site generator build process and copying the resulting static files onto a file server or CDN, it was straightforward to replace Netlify builds with Forgejo Actions provided by Codeberg. The resulting Forgejo Actions deployment config is almost identical to a Github Actions deploy job.

→ I began sharing some of my Bunny onboarding learnings as part of my Eurostack posts.

Plausible Analytics: Privacy-First Insights #

Website analytics are an integral part of understanding user engagement. Using a third-party analytics provider fits nicely into the JAMstack architecture. I replaced Google Analytics with Plausible Analytics already years ago. Based in Estonia, Plausible offers a lightweight, open-source, and privacy-focused analytics solution.

Unlike “Big Tech” analytics solutions, it doesn’t use cookies and is fully GDPR-compliant by default. This means I can gather valuable insights into my website’s performance without compromising on user privacy or worrying about complex consent banners. It’s a testament to how privacy-conscious alternatives can deliver powerful functionality.

JAMstack Migration & Service Responsibilities #

Service Responsibility	Previous Provider	New Provider	Key Benefit of New Provider
Code Hosting	Github (USA)	Codeberg (Germany)	Non-profit, open-source, privacy-friendly, community-driven
Static Site Deployment	Netlify (USA)	Codeberg (Germany)	Codeberg leverages Forgejo Actions, providing CI/CD similar (but not identical) to GitHub Actions
Static Site Hosting & CDN	Netlify (USA)	Bunny CDN (Slovenia)	European-headquartered, competitive pricing, GDPR compliant
Web Analytics	Google Analytics (USA)	Plausible Analytics (Estonia)	Privacy-focused, no cookies, GDPR compliant

In conclusion, my European JAMstack migration reflects a deliberate move towards a more privacy-conscious and open-source friendly internet. By choosing European providers like Codeberg, Bunny CDN, and Plausible Analytics, I’ve improved the performance of my projects and gained significant peace of mind.

For anyone looking to minimize their reliance on large US tech companies and embrace services with strong privacy commitments, exploring these European alternatives is a highly recommended step towards a more secure and ethically aligned online presence.

FAQ #

Why is it important to switch to European alternatives for personal projects, even if they don’t handle highly sensitive data? #

While personal projects might not always involve highly sensitive data, switching to European alternatives has several benefits beyond just immediate data privacy concerns:

Firstly, it aligns with the principles of data sovereignty, meaning data is governed by European regulations like GDPR, which are generally more stringent than those in other regions. This proactive approach can give greater peace of mind and prepare projects for potential future growth where data sensitivity might increase.
Secondly, it supports a more diverse and decentralised tech ecosystem, reducing reliance on a few dominant US-based “Big Tech” companies. This creates healthy competition and encourages the development of open-source and community-driven platforms, which can lead to more innovation and ethical practices in the long run.

Are there any performance trade-offs when switching from US-based providers like Netlify to European alternatives? #

Not really. A European content delivery network like Bunny CDN can actually outperform Netlify in terms of raw CDN capabilities due to its robust network and focus on content delivery. As of 2025, Bunny CDN operates over 100 global Points of Presence (PoPs) while Netlify leverages 16 data centers. Offering broader geographical coverage results in lower latency and faster load times.

What are potential challenges when migrating existing JAMstack projects from Netlify’s build process to Codeberg’s Forgejo Actions? #

Moving from Netlify’s built-in environment, which often offers zero-config setups for common static site generators, to a more explicit CI/CD setup with Forgejo Actions is a fundamentally different deployment approach. Challenges might include:

Environment Variables: Ensuring all necessary environment variables used in Netlify builds are correctly configured in Forgejo Actions.
Build Commands: While similar, specific commands or paths might need adjustments.
Netlify-specific Features: If a project heavily relies on Netlify-specific features like Netlify Functions, Netlify Forms, or specific redirects/headers configurations, it is necessafy to find alternatives or implement similar logic within Forgejo Actions and Bunny CDN setup.

What other European JAMstack services or categories should be considered for a fully European setup? #

More complex projects might need additional solutions for dynamic functionalities that go beyond just static site hosting and analytics:

Serverless Functions: Keep an eye out for European cloud providers offering Function-as-a-Service (FaaS) or serverless computing, like Clever Cloud (France), Exoscale (Switzerland), or specific services from IONOS (Germany).
Form Handling: Use solutions like Formspark (Estonia) or build on serverless functions (hosted on a European FaaS platform) to receive form submissions to replace services like Netlify Forms.
Authentication: When it comes to user authentication, consider Identity as a Service (IDaaS) providers based in Europe, or maybe self-hosting an open-source solution like Keycloak on a European server.
Headless CMS: It’s not exactly a “migration” from a US provider but for projects based on a Headless CMS, consider to check out some European options like DatoCMS (Italy), Storyblok (Austria), or Strapi (France - open-source, can be self-hosted).

What are the main benefits of “open-source principles” for users of providers like Codeberg, compared to proprietary platforms? #

The open-source nature of platforms like Codeberg has a lot of advantages for individual users:

Transparency and Trust: The code is publicly auditable, meaning everyone can check it for security vulnerabilities or unwanted tracking. This makes people more likely to trust it than proprietary systems, where the ins and outs are kept secret.
Community-Driven Development: Codeberg is a non-profit, community-driven organisation, so its priorities are more aligned with its users than profit maximisation. This often means features that are developed based on what the community needs and a focus on making sure it’ll last.
Longevity and Control: It is less likely to get caught in a “vendor lock-in” situation or to have service terms or prices suddenly changed. If the platform changes direction, the open-source code (theoretically) allows to fork the project and continue development independently.
Learning and Contribution: For developers, hosting code on an open-source platform can be a great way to learn from others’ code, contribute back to the platform itself, and gain experience in a truly open development environment.

Bunny Shield Review: 3 Things Need Improvement

Sat, 07 Jun 2025 00:00:00 +0200

Bunny.net launched the Bunny Shield Preview earlier this year. It’s an integrated Web Application Firewall (WAF), DDoS mitigation and rate-limiting controls solution all bundled into their Content Delivery Network (CDN) platform. I’ve been testing it on a few real-world projects over the last few weeks to see how ready it is and how easy it is to use.

Spoiler: Bunny Shield is straightforward to use and makes the most of your first experience. It’s a solid start for an accessible WAF, but there are a few key improvements needed to make it a production-ready web security product.

Here’s a quick rundown of the things I noticed and the problems I came across. It’s meant to show what’s working well now and where Bunny.net could improve their security tooling before it leaves the “Preview” stage.

What Bunny Shield Gets Right #

Initial User Experience & Setup Simplicity #

Bunny Shield carries on the tradition of Bunny.net of offering clean and usable interfaces. As always, you only need to do the bare minimum to get things done. It was easy enough to enable the service and set up firewall or rate limit rules. For smaller sites and teams without dedicated security personnel, this kind of usability is invaluable.

Default Security Coverage #

The out-of-the-box rules catalogue seems pretty reasonable to me, catching common threats without overwhelming users with complex tuning. For most users, this makes Bunny Shield a no-brainer for basic web protection.

What Bunny Shield Needs to Improve #

Certificate Renewal Blocked by WAF #

One of the most critical issues I encountered was Bunny Shield blocking HTTP-01 challenges from Let’s Encrypt. This sometimes seems to stop the automatic TLS certificate renewals through Bunny’s own certificate mechanism, which could cause service disruptions if not caught in time.

The good news is that Bunny’s support team have already said they know about the problem and are working on a fix. For now, users relying on HTTP-01 validation (e.g. by using the built-in Bunny CDN certificate management) should proceed with caution or disable the WAF.

No Bot Allowlisting for Good Bots like Ahrefs or Semrush #

Currently, Bunny Shield doesn’t offer any particular bot management solution. This means there’s no way to tell the difference between harmful bots and legit crawlers (like Ahrefs, Semrush, and other SEO tools). I saw this causing these “good bots” to be blocked, which harmed my SEO audits and (potentially) visibility.

It’d be great to add a bot management layer or at least some simple allow listing functionality for expected crawlers. The opposite is also true: I’m sure some people would love to trap rogue or misbehaving AI scraper bots in a maze.

Overall, I’d love to see Bunny hop on board with the HTTP Message Signatures bandwagon that Cloudflare recently kicked off.

Limited Logging and Visibility into WAF Events #

There’s a clear difference between the 403 traffic logs and the WAF event logs. I can see that requests were blocked, but there’s no clear traceability to what specific WAF rule was triggered and why. This makes it tricky to debug false positives or fine-tune firewall behaviour, especially when you’re dealing with a lot of data.

Missing WAF Error Code Lookup Tool #

When Bunny Shield blocks a request, it returns a 403 with a unique deny code (e.g. WAF403-XXXX). Unfortunately, there doesn’t seem to be a lookup tool for these codes at the moment. If we could get a dashboard-integrated lookup, that’d make a big difference to WAF rule debugging.

Final Thoughts: Promising Foundations, But Still in Beta #

All in all, the Bunny Shield Preview is doing really well, especially when it comes to how easy it is to use. It’s accessible, integrated tightly into the Bunny.net platform, and it’s already solving basic security needs for many developers.

That said, there are a few key things that need to be sorted for Bunny Shield to become a great product. For example, observability, bot management and configurability need to be improved.

I’m excited to see how this product develops, and I’ll definitely be keeping an eye on the next updates.

Got Feedback or Similar Experiences? #

Have you had a chance to try out Bunny Shield yet? Did you encounter similar issues? Found any workarounds yet? I’d love to hear about your experience – just send me a message on Bluesky or Mastodon.

Let’s help make Bunny Shield a secure, modern WAF that everyone can use.

Bunny CDN: How to redirect Root Domain to Subdomain

Tue, 22 Apr 2025 00:00:00 +0200

Although technically no longer necessary, the www subdomain is deeply ingrained in internet history and user habits. Redirecting root domains (like janbrennenstuhl.eu) to its www subdomain (www.janbrennenstuhl.eu) offers benefits, especially when using a content delivery network (CDN) or caring about search engine optimization (SEO).

In this article, I share three different approaches to redirect a root domain to its www subdomain when using Bunny CDN with Bunny DNS. All three approaches are technically sound and working solutions, returning proper A records for the domain root.

Option 1: Redirect Record (RDR) #

Bunny DNS supports the creation of special redirect records. While the initial resolution happens at the DNS level, Bunny automatically applies HTTP redirects, instructing the browser ultimately to make a request to the redirected location.

Configuring a new redirect record is as simple as heading over to Bunny DNS and choosing RDR as record type in the “Add DNS Record” prompt. Setting up a redirect record works as long as you don’t already have the hostname registered.

Bunny DNS – How to add a redirect record

Option 2: Pull Zone Record (PZ) #

A “Pull Zone” in Bunny CDN is a fundamental concept that defines how static assets are delivered using Bunny’s network of edge servers. A PZ record is a Bunny-custom, special DNS record type that provides some level of abstraction when glueing a domain to a “Pull Zone”. Instead of directly resolving to an IP address, Bunny DNS understands that requests for that particular domain should be directed to its CDN infrastructure associated with the corresponding “Pull Zone”.

Configuring a new “Pull Zone” record is as simple as heading over to Bunny DNS and selecting PZ as record type in the “Add DNS Record” prompt. Prerequisite is an existing “Pull Zone” that the root domain should point to. The actual HTTP redirect would then be handled by a dedicated “Edge Rule”.

Bunny DNS – How to add a pullzone record

Option 3: Canonical Name Record (CNAME) #

Finally, Bunny DNS also allows creating CNAME records for root domains. This is possible (and also returns an A record) as Bunny transparently performs CNAME flattening under the hood.

Configuring a new canonical name record is as simple as heading over to Bunny DNS and choosing CNAME as record type in the “Add DNS Record” prompt. The actual HTTP redirect would then be handled by a dedicated “Edge Rule”.

Bunny DNS – How to add a CNAME record

Dedicated Edge Rule #

In case a root domain points to a Bunny CDN “Pull Zone” either by using either a PZ or CNAME record (Option 2 or Option 3), the actual HTTP redirect (301) needs to be applied using a dedicated “Edge Rule” like this:

Action	Value	Status Code
Redirect to URL	`https://www.janbrennenstuhl.eu{{path}}`	301

Condition	Match	Value
Request URL	Any	`://janbrennenstuhl.eu/`

Bunny CDN – How to redirect Root Domain to Subdomain

Summary #

In summary, while the www subdomain isn’t a technical necessity in the modern web, redirecting root domains to it is a best practice for SEO consistency, technical flexibility, and user experience. It ensures that websites are treated as a single entity by search engines and provides a consistent and potentially more reliable experience for visitors.

Bunny.net DNS supports three different approaches for redirecting a root domain to its www subdomain. All result in perfectly fine DNS configs and high-speed resolution when paired with Bunny CDN.

Learning AssertJ: Null ain't Blank

Tue, 05 Sep 2017 00:00:00 +0200

As programmers, we probably use strings more than any other type of data structure. The cross-language String environment is characterized by shared vocabulary and well-established concepts. And it is especially important to follow these established naming conventions when it comes to null strings vs empty strings (and blank strings).

AssertJ – Blank vs Null Strings

With Apache Commons StringUtils and Hibernate Validator Constraints, two popular and widely adapted projects established a de facto definition for a blank string:

“A blank String is a CharSequence that is empty (""), null or whitespace only.”

While recently investigating a bug in one of our systems, I stumbled upon a slight divergent and unexpected string handling in AssertJ, a fluent assertions framework for Java. This AssertJ misbehaviour has the potential to lead to severe troubles – which is why I consider it to be a bug. As it turns out …

String s = null;
assertThat(s).isNotBlank();

… passes. This is definitely not expected behaviour, since (following the common definition above) a null string is considered to be a blank string and vice versa. So if you’re used to Apache Commons StringUtils and Hibernate Validators behaviour, you want to make sure that also your tests expect the correct behaviour.

Do you know further libraries that follow a different blank string definition or if you have a different opinion about all of this, drop me a message via Mastodon!

On Making Spring Security OAuth RFC-compliant

Wed, 01 Feb 2017 00:00:00 +0100

Often supporting open source software is just fixing one tiny thing you stumbled upon. However, getting ready and diving into even the smallest piece of source-code can lead to surprising results. I fixed a small HTTP header extractor for the Spring Security OAuth project recently. Here’s what happened.

I fixed something for me #

At the very least, fixing this issue has made me happier. The problem was that the Spring Security OAuth BearerTokenExtractor was not acting in compliance to RFC7230.

Every time someone sends a request with multiple, comma-separated Authorization-Header values towards an application I am currently developing the extractor would fail in doing its job. The application would simply not accept the yet valid (and RFC compliant) request.

RFC7230, Section 3.2.2 - A sender MUST NOT generate multiple header fields with the same field name in a message unless either the entire field value for that header field is defined as a comma-separated list [i.e., #(values)] …

I fixed something for someone else #

This is a guess, but if I found this an issue then we can presume at least one other person out there on the internet did. Also, being RFC-compliant usually is a good thing and every library should handle user-generated input as resilient as possible.

I learned more about Library Maintenance #

I started by making a mistake. My original fix was a custom implementation of the TokenExtractor interface that is also implemented by the original BearerTokenExtractor within my application. This however would require me to maintain and test this piece of code that is not part of my actual business logic. I would totally loose the advantages of abstraction through a battle-proven library. It also ignored the problems true essence.

That thought lead me to try again. This time I cloned the Spring Security OAuth project and tried copy-and-pasting my slim Java8 Streams API based TokenExtractor and its tests into the library. This lead to a bunch of discoveries about library maintenance — Spring Security OAuth…

… is written in Java6,
… has only other Spring projects as dependencies,
… needs to be backward compatible.

So, if you plan to support a project that is out there for a while, keep in mind to leave all your fancy tooling at home. Also try to read and follow the projects code and style guidelines before receiving requests for change in your pull-requests.

Of course, as with anything around the globe, there is even more to this than meet the eye. However, this was my resulting pull request ~~and it is waiting to get merged~~.

I learned more about HTTP Headers #

In the beginning, I learned about the diversity of HTTP Headers. Being spoiled by all these framework abstraction layers nowadays, it hit me that HTTP Headers are simply yet another interface. And we want to handle all interface interaction in the most resilient way.

Inspired by some of my colleagues, I started to delve into RFCs. I started with RFC6750 (describing the Bearer Token Usage within OAuth 2.0) and ended up in the even more fundamental RFC7230 (on Message Syntax and Routing of the Hypertext Transfer Protocol).

Now I am more familiar with the possible appearances of HTTP Headers and the work behind the Spring Security OAuth project.

~~With my fix in one of the next releases, Spring Security OAuth is going to also be able to handle requests like this:~~

GET /presence/alice HTTP/1.1
Host: server.example.com
Authorization: Basic YXNkZnNhZGZzYWRmOlZLdDVOMVhk, Bearer mF_9.B5f-4.1JqM

Update 16.02.2017: As Joe Grandja (Spring Security Engineer) argued, the user agent is supposed to choose “only 1 of the challenges (if there is more than 1 challenge) to authenticate against”, so my pull request was rejected. Feel free to read all the comments in the corresponding issue for all the details.

The simplest piece of work can lead you anywhere #

I just wanted to make the Bearer Token extraction more resilient. Yet, this lead to a ticket and a pull request, a rabbit hole of discovery about HTTP Headers and open source library maintenance and the motivation to write a blog post about it. Ultimately, I did get what I wanted in the first place though: More fun during work.

Open source is a lot of things to a lot of people. To me, it’s a way to learn while I help, however small the task may seem. I’d love for you to share with me on Twitter what open source means to you.

This post was heavily inspired by Phil Nash.

AWS DynamoDB: Backup & Recovery Strategy

Mon, 18 Jan 2016 00:00:00 +0100

“Amazon DynamoDB is a fully managed NoSQL database service […]”. That’s what Amazon states in the DynamoDB FAQs. However, their definition is not quite what I would understand as fully managed. It’s missing counter-measures against the human error. For that reason, we need a DynamoDB backup and recovery solution, our DynamoDB safety-net…

DynamoDB Threat Scenario #

Imagine your entire AWS infrastructure, including your DynamoDB tables, is defined and set up via AWS Cloudformation (btw, make sure to read why Cloudformation sucks) or even worse by hand. Further imagine you actually use your tables to store more or less important data. And finally imagine someone deleting your tables and/ or stacks by accident or your software behaving somehow unexpected, dropping tables. See?! DynamoDB does not ship with built-in backup and recovery implementations, so we had to set up our own protective strategy.

The solution we came up with basically orchestrates AWS DynamoDB with Datapipelines, S3 and Amazons Lambdas to establish not just a daily backup mechanism but also to allow simple staging (production to testing environment) and recovery possibilities. This is how it works:

DynamoDB Backup Components

DynamoDB w\ AWS Datapipeline & Lambda #

In addition to the table definitions, our Cloudformation-based infrastructure setup also describes Datapipelines for the tables in our production environment which nightly backup DynamoDB tables to S3. Furthermore, these Datapipelines’ ObjectCreated-Events trigger a small AWS Lambda function we also have in place, which is responsible for deletion and recreation of the corresponding tables in our testing environment. Separated in time, additional Datapipelines refill these empty tables with the productive backups created earlier.

In that way, we established a battle-proven backup and staging solution for AWS DynamoDB. But is this of any help when we have to deal with a real DynamoDB disaster, e.g. accidental table deletion or data corruption? Yes, it is:

Due to the infrastructure as code approach, we can easily recreate entire stacks of DynamoDB tables and Datapipelines (yes, we run tables and pipelines in separated stacks) and use our latest backups to recreate a stable state. Everything with just two or three commands.

However, we don’t have any automation in disaster recognition yet as in my eyes it’s quite hard to identify possible disasters and (especially) distinguish those from intentional table changes. Also, with this setup we don’t have any chance to properly react on DynamoDB (region, AZ, etc.) not reachable/ available issues.

Further downsides: Our approach is heavily dependent on Datapipelines (I already have a ‘Why I hate AWS Datapipelines’ post in my pipeline ;D) and due to the orchestration of four Amazon Web Services it is not as simple nor cost-efficient as I would want a proper solution for this common problem to be.

Hope you got some inspiration for your future DynamoDB setup. If you have any questions or improvement suggestions make sure to leave a comment or contact me directly.

Evolution of a Microservice: A Git History Visualisation

Thu, 21 May 2015 00:00:00 +0200

Microservices have revolutionized the way we develop and deploy software applications. But with great power comes great complexity. As microservices become more and more ubiquitous, developers must grapple with new challenges, such as managing dependencies, ensuring scalability, and maintaining consistency across services. In this article, I take a deep dive into the evolution of a microservice, from its inception as a small, independent module to its growth into a complex kraken…

Following the common pendulum theory, every tech debate oscillates between at least two currently trending topics. The same is true when it comes to micro-services (vs. integrated systems). One of my former teams decided to follow a radical microservices approach, favoring the benefits of smallest possible deployment units over integration (and its advantages) and preferring infrastructure as well as communication complexity over tight coupling. Why we chose to do so and how we make sure to keep our services micro, I’m going to explain in more detail…

Microservices, Self-contained Systems & Integrated Applications #

Back then my team was responsible for parts of one a core selling flows. With thousands of customers-to-be every day and direct major business impact, we not just need to be reliable as well as always up-and-running, but also fairly flexible and (even more important) fast, allowing product improvements, bug fixes and multi-variant tests to be rolled out in near-realtime. Helping us to achieve those goals are our continuous live deployment (CLD) infrastructure, our near-realtime monitoring as well as alarming setup and most importantly our constantly evolving, stateless micro-services…

The micro-services architecture I am referring to in this article is the probably most remote antipole of what is commonly known as monolithic, integrated system (or what some companies call legacy, heritage or mothership). Between those two extrema, there is a whole bunch of manifestations, variations and adoptions like the so-called Self-contained System (SCS) architecture. However, I am usually not a big fan of some technical middle courses as in my experience they often combine not the intended advantages but the downsides of two radical different ideas. Furthermore, if a framework (and an architecture is nothing else) allows people to implement their first gut feelings instead of understanding the greater ideas, concepts and reasons, they tend to do so.

On the other hand, sticking with and optimizing your infrastructure for one clean approach (not in a religious but explaining manner) generally helps creating easy repeatable and comprehensible software architecture - as long as you can handle possible drawbacks. Having implemented a strict microservices-based deployment architecture in our core business, we now benefit from a couple of great possibilities even our business people find it great to work with:

Watch the accompanying video on YouTube (external site, not embedded).

Microservices, Don’t let’em grow! #

We have tiny, decoupled applications that are deployed independently. Due to their small footprint, each of them is easier to understand (even by new colleagues) and can be cloned and built in almost no time, which speeds up not just our dev productivity but also our deployments. Additionally, our microservices enable us to exchange parts of our tech stack pretty easy and make us in that way eager to try out new things. Whether it’s about server-side vs. client-side rendering, SQL vs. NoSQL databases, Play vs. Spring or any other major design decision, we don’t need to compromise, we can use whatever fits best and helps us to solve the microservices single purpose. It’s a great feeling and even better: It helps us reducing complexity and dependencies in each of our deployment units. However, to be able to enjoy these advantages you need to be beware of its drawbacks…

In my eyes the major challenge of a microservices architecture is not that shifted complexity of its distribution, but that you need to cut. It’s a big drawback, because every software tends to grow bigger - it’s just a matter of time until a single-purpose microservice becomes a multi-purpose legacy app. There’s no way to prevent your software from doing so. Whether you cut your services functional or by following business use-cases, you have no other chance than to be fully-aware of that fact, and you need to cut carefully and (most importantly) you need to cut often.

The video above visualizes the evolution of one of our microservices on a source-code level. As almost every green field software, it starts fairly simple but quickly outgrow its originally intended size and (even worse) its single-purpose approach. Although fighting back with more or less effective refactorings, the service grew from 5k up to 15k lines-of-code (LoC) within a year until we started to actively enforce our microservice-policy. As the Gource visualization shows, we started to decouple our service’s back- and frontend functionality by migrating from server- to client-side rendering and by introducing a separate, Grunt-based build-process for our frontend-resources (see 01:30).

Next, we where able to identify further decoupling possibilities more easily and started moving major backend functionality to a new, separately created microservice, shrinking down the code-base of this particular service (which basically serves a couple of public webpages) back to round about 5k LoC (see 01:50). Being done with that, our service now is in a state where we can directly benefit from all the advantages we discussed before. Furthermore, we ended up with a clean decoupling between mainly enduser-centric code and a more business-administration REST-API.

tl;dr - Take a radical approach! #

When it comes to tech decisions, going all-in and optimizing for one strict approach (one pole of the spectrum) and being aware of its possible drawbacks is always better than mixing contradictory ideas, patterns or philosophies. Although in theory, combining contradictory approaches might sometimes sound like a great idea (“hey, let’s combine the best of’em all!”), in practice they’re usually not.

Whether it’s about server-side vs. client-side rendering, serverless vs. servers-as-cattle, or the red vs. blue pill, you want to settle for exactly one extrema. However, you need to know and understand the drawbacks. Establishing and maintaining e.g. an microservices architecture is continuous work. Evaluating the actual size as well as purpose of your services (and service landscape) is essential and cutting down your services regularly is a must if you want to benefit from your architectural decision for microservices.

Abusing JSON: Why AWS CloudFormation sucks?!

Thu, 21 May 2015 00:00:00 +0200

AWS CloudFormation (CFN) is a file-based interface for provisioning AWS resources and in that way is Amazons answer on the ‘Infrastructure as Code’ paradigm. I am a huge fan of this approach and CFN is a big step into the right direction as it supports not just common DevOps ideas, by allowing devs to create, update and delete the operational infrastructure they need with their own tools (a text editor, version control and tests).

It also enables you to think different about your IT infrastructure and how to work with it (e.g. just think about immutable stacks). However, I see some issues that become more important if you work with CloudFormation on a daily basis …

JSON Lang, WTF?! #

CloudFormation templates are simple JSON-formatted text files that (in theory) can easily be created and maintained by hand as nowadays every dev (and/ or ops) should be able to read and write valid JSON. However, AWS CloudFormation uses the originally static lightweight data-interchange format as a scripting language, allowing e.g. the definition and usage of parameter and (somehow crazy) intrinsic function references.

In my eyes, this is a heavy misusage that leads to several problems regarding complexity and comprehensibility. You can find several deterrent examples that illustrate this right in the official AWS developers guide. If you still don’t think JSON is a syntax-bloated, hard-to-read data format just have a look on how to define DataPipelines using CloudFormation and tell me you like it and that you are able to easily understand all the definitions as well as their interactions. It took me some time…

But that’s not even the worst part. JSON also lacks some feature substance that some devs would regard heavily essential. In contrast to its technical superset YAML, JSON is not aware of e.g. comments or user defined data types. In my eyes, this is a huge downside and a massive argument against Amazons classical CloudFormation design.

All in all I don’t quite get the whys and reasons which made Amazons engineers believe that the abuse of JSON for scripting would be a better and more innovative idea than go for something like an established scripting language like Python or JavaScript.

But enough JSON-bashing for now. Let’s solve at least some of our AWS CloudFormation issues.

CFN-Sphere #

Currently there are a couple of tools around that sit on top of CFN as an abstraction layer and offer not just a YAML-based config approach, but also further functionality that comes in handy if you work with infrastructure as code during your everyday business.

One of those tools is CFN-Sphere, a lightweight Python-tool that aims to simplify AWS CloudFormation handling not just for infrasturcture automation but also for human interaction. Initially built by Marco Hoyer (a former colleague and friend of mine) CFN-Sphere comes with fundamental support for e.g. cross stack parameter references and automatic stack dependency resolution including circular dependency detection. Both are amazing features also for smaller AWS infrastructure definitions like the backup and staging environment for DynamoDB tables I am currently working on (More on that topic in one of the upcoming posts).

So if you are struggling with your with your CloudFormation config or if you have a certain feeling that CFN files having around several hundred lines of code are almost unmaintainable, make sure to check out CFN-Sphere and its stack template feature. And help making it even greater by becoming a contributor!

Python Docopt: Convert Tuple List to Dictionary

Tue, 28 Apr 2015 00:00:00 +0200

Using Python Docopt (for beautiful command-line interfaces) I recently had the problem of converting a list of tuples into a dictionary. In this post I’d like to document the solution…

"""
usage:
 mytool [options]

options:
 --foo-bar=PARAM1,PARAM2,... params to do stuff with
 [default: key1.value1,key1.value2,key2.value2]
"""

from docopt import docopt
from collections import defaultdict


def run():
 params = [tuple(param.strip().split('.')) for param in arguments["--foo-bar"].split(",")]
 params = convert_tuple_list_to_dictionary(ignored_resources)

def convert_tuple_list_to_dictionary(list_of_tuple):
 dictionary = defaultdict(list)
 for k, v in list_of_tuple:
 dictionary[k].append(v)

 return dictionary

if __name__ == "__main__":
 run()

This code is able to convert the parameter value string key1.value1,key1.value2,key2.value2 into the following dictionary:

{
 'key1': ['value1', 'value2'],
 'key2': ['value2']
}

If you know a leaner solution to solve this problem, let me know!

How to Migrate SVN to GIT with History?

Wed, 15 Apr 2015 00:00:00 +0200

Today I’d like to share my solution for a common problem a lot of IT companies experience when finally moving from Subversion (SVN) to Git as their central version control system (VCS): Converting a SVN repository into a Git repo including its commit history and authors.

To achieve this, I decided to split this issue into smaller pieces:

Extracting all authors of the SVN repository,
Creating a proper authors file mapping SVN users to Git users,
Converting SVN repo to Git including its history and authors.

Here is what I did:

Converting SVN to GIT w/ History & Authors #

Checkout your SVN repository.

svn co SVN_REPO_URL

Go into the repo directory and extract authors from the SVN repo using this statement.

svn log --xml | grep author | sort -u | perl -pe 's/.>(.?)<./$1 = /'

Using this information, create an authors.txt file like this:

# SVN_AUTHOR_HANDLE = GIT_AUTHOR_HANDLE <GIT_AUTHOR_EMAIL>
jbrennenstuhl = janbrennenstuhl <xyz@brnnnsthl.eu>
...

Using this file you now can start converting your SVN repo into a Git repo.

git svn clone -A authors.txt SVN_REPO_URL

This will convert the old SVN repo into a new Git repo including the complete SVN history and mapping its authors following the mapping in authors.txt. Depending on the dimension of the SVN repo history, this can take several minutes or even hours!

If you don’t want to have the entire SVN history you could choose to start from a specific revision like this:

git svn clone -rSVN_REVISION:HEAD -A authors.txt SVN_REPO_URL

Still using SVN? Start converting to Git now!

How to Check Github Licenses

Wed, 25 Mar 2015 00:00:00 +0100

On Github, every organization with a fast growing open-source codebase quickly figures out a couple of shortcomings regarding the Github web interface. Especially, when it comes to Github organizations that are already around for some time, you often find some information missing that could probably help managing and controlling your organizations repositories…

Github & Licenses #

For months now, Github addresses in particular the often discussed license-problematic by implementing the License Picker feature and introducing the choosealicense.com information page. That’s great if you start with new projects and repositories, but what’s missing is some kind of license overview and maintaining functionality that allows dealing with licenses of already existing repos.

For that reason, I just built a small Python tool to help you getting an overview of your repos licenses: Github-Licker, a Github License Checker!

Github License Checker #

Using the Github API v3 including the Licenses API developer preview, Licker is able to check licenses of all repositories in your Github organization. It searches all your own repositories (no forks!) and especially helps identifying repos without license files.

pip install github-licker
licker -h

usage:
 licker [TOKEN] [options]

options:
 -o, --organisation=ORGANIZATION the Github organization you want to check [default: ImmobilienScout24]

Github-Licker itself is licensed under Apache License, Version 2.0, for what reason you can easily help me to improve it!

AWS Monocyte: Let's Build a Cloud Immune System

Wed, 18 Mar 2015 00:00:00 +0100

With Ireland and Frankfurt being available as AWS regions nowadays, Amazon (more or less) tried to extinguish EU and especially German legal concerns regarding storage and processing of privacy-related data. However, for European companies it remains difficult to prevent (accidental) usage of services and resources outside the EU, as there is still no standardized way to restrict AWS-account rights on this region-level. Especially in open, DevOps-inspired company cultures this becomes a major issue:

AWS & YBIYRI #

The majority of European companies my colleagues and I recently interviewed about their AWS infrastructure, implemented (or plan to implement) some kind of multi-account setup, meaning that single product teams own their own accounts (which usually are aggregated via consolidated billing). One reason for this decision is (in addition to hard AWS API limits) that they try to implement the so-called “You built it; You run it”-approach, which is also propagated by Amazon. This trending approach implies that every AWS account is managed entirely and autonomously by its corresponding development team, which eventually is not anymore just responsible for building software components and products but also becomes responsible for operating the required infrastructure.

This (of course!) leads to several conflicts of goals. On the one hand companies often want their teams to work with AWS independently and manage their own accounts mostly autonomously. On the other hand they are usually bound to legal frameworks (e.g. the EU Data Protection Directive and/ or German privacy laws) and for that reason want to search and destroy non-EU AWS resources relentlessly.

A Cloud Immune System #

To accomplish that, we built our own base-layer of what we call a cloud immune system: AWS Monocyte. The name Monocyte is related to a type of white blood cells that are part of e.g. a human’s innate immune system, the first line of defense being responsible for searching and destroying alien organisms to prevent unwanted infections.

As Arne Hilmann and I explained during the AWS UserGroup Meetup this March at the Immobilien Scout HQ in Berlin, our tool is (in that biological sense) able to find and delete AWS resources in regions outside the EU. This allows companies to make sure that (accidentally) created non-EU resources are destroyed on a regular basis and helps in that way to (unfortunately reactively) enforce legal requirements.

Right now, AWS Monocyte is able to handle just a subset of the plenty Amazon Services, including Cloudformation Stacks, EC2 Instances, EC2 Volumes, RDS Instances, RDS Snapshots, DynamoDB Tables and S3 Bucket. Implementing support for further Amazon Web Services is pretty easy and I hereby ask you to fork our project and help extending its power!