Modern Platform Engineering: Open Source

Open Source Platform Engineering

As a platform engineer, the build vs. buy debate can be exhausting, especially with a software engineering background, as it often leads to a desire to create and build your own platform. Numerous options exist for fully managed, black-box platform engineering solutions that promise a single pane of glass for all your platform needs. However, I enjoy challenges and embrace complexity, which is why I prefer to go the open-source route and build things myself.

I believe the best learning experience for any platform engineer comes from constructing a platform from scratch. The thrill lies in utilizing available open-source tools, mixing and matching them, and discovering the most optimal ways to blend them together. While this approach undoubtedly adds complexity and numerous challenges, isn’t that what life is all about?

Over the years, I have worked with a variety of open-source tools and technologies, some excellent and some not so great. Here’s a curated list of open-source tools that I recommend if you’re currently building anything.

If you need an intro to platform engineering, please read my previous blog: Platform Engineering 101

Open Source Platform Engineering Tools & Components

Developer Control Plane:

• GitLab: A complete DevOps platform for CI/CD, version control, and project management.
• VSCode: A lightweight code editor with extensions for any language or framework.
• AppFlowy: An open-source alternative to Notion for project and document management.
• Backstage: A developer portal to simplify building and managing services and infrastructure.

Integration & Delivery:

• Terraform: Automates infrastructure management with Infrastructure as Code (IaC).
• Helm: A package manager for Kubernetes, helping manage Kubernetes applications.
• ArgoCD: Continuous delivery for Kubernetes, syncing applications with Git repositories.
• Harbor: A container registry for storing, scanning, and managing Docker images.

Security & Identity:

• Trivy: A security scanner for containers, Kubernetes, and codebases.
• Gitleaks: A tool to scan codebases for secrets like API keys or credentials.
• SonarQube: A tool for inspecting code quality, vulnerabilities, and maintaining code health.
• OpenPolicyAgent (OPA): A policy engine for enforcing access control across your stack.
• Keycloak: An identity and access management solution with SSO, MFA, and more.
• Hashicorp Vault: Securely stores and manages sensitive data like secrets and encryption keys.

Monitoring & Logging:

• OpenTelemetry: A standard for collecting metrics, traces, and logs for performance monitoring.
• Grafana: A platform for visualizing metrics with beautiful, shareable dashboards.
• Loki: A log aggregation system integrated with Grafana for efficient logging.
• Prometheus: A monitoring and alerting toolkit that scrapes metrics from apps and services.
• Jaeger: Distributed tracing to optimize microservices performance.

Infrastructure:

• OpenStack: A cloud computing platform to build and manage public and private clouds.
• Kafka: A real-time data streaming & messaging platform for high-throughput, low-latency pipelines.
• Kubernetes: An orchestration tool for automating deployment, scaling, and managing containers.
• Nginx: A web server and reverse proxy for scaling apps with load balancing and caching.
• OPNSense: An open-source firewall and router platform for managing network security.

Testing:

• Grafana K6: A modern load testing tool to ensure apps handle scale and performance demands.
• Playwright: A tool for automating browser testing across different browsers and devices.
• Appium: Automates mobile app testing for Android and iOS.

Workflows:

• n8n: A no-code automation platform for creating workflows between different apps and services.
• Airflow: An orchestrator for scheduling and managing complex workflows and data pipelines.

Artificial Intelligence:

• LLaMA: Meta’s open-source large language model for AI research.
• AutoGPT: An AI project that chains GPT tasks for self-prompting AI systems and developing autonomous AI agents.
• FlowiseAI: A UI for building GPT-based workflows for creating intelligent agents without code.

Honorable Mentions:

• AWS CDK: The AWS Cloud Development Kit allows developers to define cloud infrastructure using familiar programming languages.
• Crossplane: Crossplane is an open-source project that enables you to manage cloud infrastructure and services using Kubernetes.
• FluxCD: FluxCD is a set of continuous delivery tools for Kubernetes that help you automate the deployment of your applications using gitops.
• Mermaid.js: Mermaid.js is a JavaScript library that lets you create diagrams and visualizations using a simple Markdown-like syntax.
• PostGresSQL: PostgreSQL is an open-source, advanced relational database management system known for its scalability, extensibility, and support for complex queries and data types.

To wrap things up, utilizing open source tools for building your platform is an excellent way to reduce upfront costs while gaining comprehensive experience. The freedom to customize and adapt these tools can lead to fantastic gains in efficiency and collaboration. That said, it’s essential to keep in mind the challenges that come with them — like added complexity, the need for ongoing maintenance, long term costs, security vulnerabilities and deprecation of tools.

If you’re part of a small team or a solo engineer, a managed solution might be the best choice. However, if you have a large, experienced team, a managed solution might still be the best choice. Just kidding! If you’re like me, you’ll likely want to experiment with these fantastic open-source tools and see if they can meet your needs. The key is to start small but always think big.

I hope you enjoyed this article! Please note that it reflects my personal, somewhat biased perspective.