A Modern Data Platform for Better Investment Decisions

“This project was more to me than just a technical implementation – it was, above all, one of the first major projects where we didn't just deliver a solution, but built real trust. We started out with a clearly scoped use case, but it quickly became clear that this was actually about much more – data, processes, governance, and ultimately how decisions get made.” — Georgi Pargov, Project Lead and Analytics Engineer

With every step, it became apparent that the platform wasn't just solving an existing problem, but opening up new possibilities. That was exactly the moment a project turned into a long-term partnership.

Before the project began, our client's core investment processes were still heavily Excel-based.

Our solution: build a scalable data platform on Databricks – including automated pipelines, centralized data storage, governance with Unity Catalog, and interactive decision support via Streamlit. We designed the data structure so it could be used directly for advanced analytics and AI use cases, without requiring further preparation.

The result:

– Single source of truth for investment data

– Faster, consistent analyses and reports

– Reduced operational risk

– Greater autonomy for investment teams in scenario analyses and portfolio simulations

– A structured platform for future AI analyses and test runs

– A long-term, sustainable platform that scales effortlessly to future requirements

To achieve this result, we structured the process into 3 steps.

Step 1: Moving away from Excel – building a scalable foundation

The first phase focused on migrating core investment workflows from Excel into Databricks. At that point, Unity Catalog wasn't yet available in its current level of maturity, so the focus was on delivering value quickly: automated pipelines, centralized data storage, and reproducible analyses.

This step alone delivered immediate value:

– A central single source of truth for investment data

– Less manual effort for investment teams

– Fast access to consistent, up-to-date information

– Reduced operational risk in day-to-day reporting

Most importantly, though: investment managers could spend less time reconciling numbers and more time interpreting them.

From the outset, though, we treated the platform as an evolving capability – not a one-off IT project. We continuously tracked technological developments and new platform features to keep the solution future-proof over the long term.

Step 2: From functional to enterprise-grade – governance with Unity Catalog

With the introduction of Unity Catalog as the standard for governance, security, and data lineage in Databricks, we deliberately evolved the architecture further.

The initial setup without Unity Catalog was functional, but it also came with certain limitations:

– no centralized governance across teams and workspaces

– limited access control for sensitive investment data

– lack of transparent traceability of data usage

That's why we migrated the entire platform to Unity Catalog and introduced a clear environment strategy, fully integrated into Azure DevOps for versioning and controlled deployments: DEV → TEST → PRP (Pre-Production) → PRD (Production)

This step was not just a technical advancement, but a key building block for risk and compliance requirements.

Business impact:

– Clear separation between development and production

– Improved auditability and compliance

– Controlled access to sensitive investment and risk data

– Lower risk when introducing new analytics capabilities

For a regulated, globally operating organization, governance isn't overhead – it's the foundation for trust, scalability, and sustainable data-driven decisions.

This governance layer created the stable foundation needed to rethink how insights are delivered.

Step 3: From dashboards to decision support – putting Streamlit to work

With data pipelines and governance in place, the next challenge came into focus: the user experience.

The existing solution was built on Databricks dashboards. That works well for static reporting, but it hits its limits as soon as business users want to analyze scenarios, adjust assumptions, or work with data interactively.

For this reason, we introduced Streamlit as a new interface layer for investment analytics.

Streamlit is a modern framework for data-driven applications – not just for visualization.

As a result, the way investment managers work has fundamentally changed:

– Interactive applications instead of static dashboards

– Custom workflows for scenario analyses, stress tests, and portfolio simulations

– Fast iteration: new requirements can be implemented within days

– Tight integration with Databricks, keeping data logic and user interaction close together

Business impact:

– Faster adaptation to market and regulatory requirements

– Tools that reflect real investment processes

– Greater independence for business units in analyzing and evaluating data

One thing became clear to us: classic BI tools like Power BI remain essential for standardized reporting. Streamlit, on the other hand, plays to its strengths in decision support, shifting the focus from “What happened?” to “What should we do next?”.

Enabling the Business: analytics tools for business users

Technology only creates value if it's usable.

To reduce dependency on technical teams, we developed our own Python packages that make working with data simpler:

– Standardized functions for reading and writing Databricks tables

– Simple helper functions for analyzing datasets

– Reusable building blocks for common investment analyses

This abstraction layer reduces technical complexity while ensuring consistency and data quality.

Business impact:

– Faster time-to-insight

– Less friction between IT and investment teams

– Lower operational risk through standardized access

– Greater independence for business users

Overall impact: beyond technology

This transformation created measurable value for everyone involved:

For the investment teams:

– Fast access to trustworthy data

– Interactive tools for real decision-making processes

– Less manual effort

For the organization:

– Stronger governance and regulatory assurance

– A scalable foundation for future use cases

– Lower risk with future development

– Higher development speed alongside a consistent level of control

Our role went beyond pure implementation. We actively advised on architecture, governance, and operating models, and enabled internal teams to keep developing the platform on their own.

The result is not just a modern data platform, but a sustainable capability within the organization: a foundation that makes it possible to make investment decisions faster, with greater confidence, and on more solid ground.