Table of Contents
An Introduction to Microsoft Fabric
Microsoft Fabric is a Software-as-a-Service one-stop-shop for analytics that provides organisations with a comprehensive and streamlined analytics platform from data ingestion to Business Intelligence and actionable insights.
During the past few years, complexity and fragmentation have often been significant obstacles for organisations striving to derive valuable insights. The process of integrating various products, managing security, and ensuring seamless data flow can consume valuable time and resources.
Microsoft Fabric is now addressing these challenges head-on by integrating some Azure and open-source capabilities within its Power BI ecosystem (including Azure Data Factory, Azure Synapse Analytics, Apache Spark, and Delta Lake). This represents the future of analytics.
Microsoft Fabric has been under development for a few years. It is the SaaS evolution from Azure Synapse Analytics (PaaS). Microsoft Fabric introduces the concept of a unified SaaS fabric, similar to Microsoft Office, where each tool has an optimised experience but shares a common foundation. Microsoft Fabric offers optimised tools for different analytics tasks while providing a shared foundation for onboarding, workspace, storage, security management, collaboration, and compliance.
This product is in public preview.
What are some of the challenges that Microsoft Fabric is trying to solve?
Some of the challenges being addressed with Microsoft Fabric are:
- The complexity of architectures due to the advancements and innovations in the field of data, AI, analytics, and databases.
- Fragmentation of data due to the increase in the volume of data stored in various systems.
- Multiple products that require extensive integration and multiple teams and skills to manage them.
- Security management, having multiple products means that security has to be managed across all of them.
- Complex cost management, many organisations have migrated to the cloud during the past few years, and most PAYG (pay-as-you-go) models are difficult to understand.
What are the benefits of Microsoft Fabric?
Some of the benefits offered by Microsoft Fabric are:
- One product, one experience, one architecture, one business model.
- Software-as-a-Service offering: Microsoft takes care of everything, from the application to networking.
- Collaboration space: bringing different professionals to share resources and the same workspace.
- Reduces the need to copy data for different solutions with OneLake.
- Low-code/no-code experience: empowering business users so data practitioners can focus on complex tasks like machine learning and data science projects.
- Reduces pricing model complexity by introducing universal computing power.
- All-round data analytics platform offering mature capabilities from Azure such as Azure Data Factory, Data Explorer and Apache Spark but built in Microsoft Fabric as a SaaS offering.
- Embraces open-source file formats Parquet and Delta Lake.
- Timely actionable insights by simplifying the architecture.
BUSINESS CASES
Some benefits include consolidating information from multiple systems into a single repository to allow business stakeholders to:
- Make strategic decisions that ensure the financial viability of the organisation
- Be compliant with government regulations
- Create predictive machine learning models
- Enable employees to be more efficient by offering operational information imported from disparate systems
- Provide real-time customer insights
What are the key components of Microsoft Fabric?
Microsoft Fabric offers the following components:
- Data Engineering – Leveraging the extensive capabilities of Spark, data engineering in Microsoft Fabric provides premier authoring experiences and facilitates large-scale data transformations. It plays a crucial role in democratising data through the lakehouse model. Moreover, integration with Data Factory enables the scheduling and orchestration of notebooks and Spark jobs.
- Data Factory – Microsoft’s Azure Data Factory is a powerful tool that combines the simplicity of Power Query with Azure Data Factory’s scale. It provides over 200 native connectors for data linkage from on-premises and cloud-based sources.
- Data Science – The data science capability in Microsoft Fabric aids in building, deploying, and operationalising machine learning models within the Fabric framework. It interacts with Azure Machine Learning for built-in experiment tracking and model registry, empowering data scientists to enhance organisational data with predictions that business analysts can incorporate into their BI reports, thereby transitioning from descriptive to predictive insights.
- Data Warehouse – The data warehousing component of Microsoft Fabric offers top-tier SQL performance and scalability. It features full separation of computing and storage for independent scaling and native data storage in the open Delta Lake format.
- Real-Time Analytics – Observational data, acquired from diverse sources like apps, IoT devices, human interactions and more, represents the fastest growing data category. This semi-structured, high-volume data, often in JSON or Text format with varying schemas, presents challenges for conventional data warehousing platforms. However, Microsoft Fabric’s Real-Time Analytics offers a superior solution for analysing such data.
- Power BI – Recognised as a leading Business Intelligence platform worldwide, Power BI in Microsoft Fabric enables business owners to access all Fabric data swiftly and intuitively for data-driven decision making.
- SQL Databases: Microsoft Fabric SQL databases are a highly transactional offering within the Microsoft Fabric ecosystem, built on one of the best relational database management systems (RDBMS) available on the market, leveraging the Azure SQL Database engine.
- Data Activator: Fabric Activator provides a no-code solution within Microsoft Fabric, designed to automate actions based on patterns or conditions detected in changing data. It continuously monitors data in Power BI reports and event streams, triggering actions when specific thresholds are met or patterns are matched. These actions can include alerting users or launching workflows through Power Automate, making it a powerful tool for streamlining responses to dynamic data changes.
What’s the goal of Microsoft Fabric?
The goal of Microsoft Fabric is to create a fast and efficient analytics platform that enables end-to-end analytics from data collection to business insights as a Software-as-a-Service offering.
Microsoft Fabric enables better integration with other services, making data more accessible to business users, and using AI to increase productivity.
What are the technical advantages of Microsoft Fabric?
There are many advantages to Microsoft Fabric:
- Low-code platform: it won’t be a surprise to see Co-pilot helping developers write the code for solutions.
- You can develop any data analytics solutions, no matter the volume or format of the information. You can build a data warehouse, data mesh, data lakehouse, and datamart, as well as operational or strategic reports without the burden of having to copy the information multiple times.
- You will be able to copy data using Spark jobs and Python notebooks and then serve the information through a data warehouse interface. Users can easily create Power BI reports in seconds, without having to duplicate the data.
- One of the best-performance databases in the world that takes advantage of the former Synapse Serverless capabilities is built-in Power BI as a SaaS offering using Direct Lake mode.
- Microsoft Fabric introduces “shortcuts”, which allow users to reference data from one place to another without moving or copying it.
- Integrate multiple data services without having to copy data multiple times. Additionally, integrate with Microsoft Teams, and Office 365 products like Excel and PowerPoint and keep your information secure with sensitivity labels.
Intelligent Data Foundation
What are the key features of Microsoft Fabric?
Microsoft Fabric has four key aspects:
- Complete Analytics Platform
- OneLake: SaaS Lake-Centric and Open Architecture
- Empower Every Business User through Microsoft products like Power BI and Office
- AI Powered
Complete Analytics Platform
Microsoft has taken notice of previous challenges and aims to transform its individual products into a unified analytics fabric (SaaS offering), facilitating faster and more efficient analytics workflows. Microsoft Fabric intends to empower organisations with an end-to-end analytics solution, allowing them to focus on the crucial aspects of analysis and insight discovery.
Microsoft Fabric adopts a lakehouse architecture, combining data lakes and data warehouses, catering to users who prefer Python, Spark, SQL, or other tools. One of the emphases is to use open formats like Parquet and Delta Lake, allowing seamless integration between different engines and toolsets. Microsoft Fabric introduces a “see-through” mode in Power BI, enabling real-time querying of data directly from the lake without data movement.
Microsoft Fabric places significant emphasis on pervasive security and governance, ensuring end-to-end visibility and control over the analytics environment. Microsoft achieves this through features like lineage passing, Microsoft Information Protection integration, and a centralised admin portal. The inclusion of these capabilities enhances data governance, compliance management, and data security, ensuring that organisations can trust the system’s reliability and meet their regulatory obligations.
Microsoft aims to provide a suite of analytics tools optimised for various roles/personas within the organisation. Including, but not limited to:
- Data Scientists
- Data Engineers
- Power Users
- Power Business Users
- Business Analysts
- Power BI Developers
- Business Intelligence Developers
- Data Warehouse Professionals
This unified approach streamlines onboarding experiences, workspace interactions, security management, compliance, and collaboration, while still offering tailored experiences for business intelligence analysts, data scientists, and data warehousing practitioners.
OneLake: Lake-Centric and Open Architecture
To address the issues surrounding data fragmentation, Microsoft introduces a lake-centric and open architecture (rather than proprietary ones, to encourage accessibility and interoperability) in Microsoft Fabric. While each tool within Microsoft Fabric caters to specific requirements, they all share a common foundation (OneLake).
OneLake, similar to the simplicity and convenience offered by Microsoft’s OneDrive, provides a unified SaaS data lake for the entire organisation. This extracts disparate storage locations into a single logical lake (OneLake) and a single copy of data (OneCopy), allowing individual departments or units to manage their data while facilitating easy data discovery, sharing, and collaboration across the organisation. Additionally, the project adopts a lakehouse architecture, enabling direct querying against the lake using SQL, while leveraging open formats like Parquet and Delta Lake. Furthermore, a SQL data warehouse built on the same open formats offers choice and flexibility to data warehousing practitioners.
OneLake provides data unification and advanced security. In traditional systems, security is implemented at the data warehouse layer, which necessitates running all data engines through the security layer at the top, causing potential efficiency issues. In contrast, OneLake’s security measures are integrated into the data lake itself, which is a more efficient solution.
OneLake introduces Shortcuts. Shortcuts enable efficient data sharing among users and applications in an organization, removing the need for data duplication. They help blend data from different business areas into a customized virtual product, catering to specific user needs. A shortcut acts as a link to data located anywhere – within or across workspaces, within OneLake or externally in ADLS or S3, making it seem like the data is stored locally.
OneLake comes with the same APIs that users are familiar with from Azure Data Lake Gen2, allowing a seamless transition for those accustomed to using Azure tools. Different compute engines work natively with the data in the lake and users can choose SQL or Spark. They can also work with the data end-to-end.
Changes applied in OneLake are made available to compute engines. For example, if a data quality issue is found and resolved using Spark, it will be then reflected in the SQL database. The unification of data and the development experience is a significant improvement over existing systems, where choosing one system locks developers into a proprietary format.
Microsoft Fabric also offers integration with other platforms like Power BI, allowing users to leverage data in a lake house or warehouse directly. It operates in Direct Lake Mode, providing the performance of import mode but without latency issues or duplicated data. Furthermore, it significantly reduces costs as it doesn’t need to send SQL queries; instead, it works directly with OneLake.
In summary, OneLake offers a unified data storage format, eliminating the need for ETL jobs into the data warehouse, and managing data loads into the Power BI data set. It optimises data performance and management, offering significant cost reduction, latency reduction, and enhanced data access and security.
Empower Every Business User through Microsoft products like Power BI and Office
Microsoft Fabric aims to empower end users and democratise analytics within organisations.
Deep integration with Microsoft Power BI and Office enables seamless data exploration, discovery, and analysis. Users can access data directly in Power BI, Excel, and PowerPoint, leveraging the benefits of a unified analytics fabric and open formats without the need for data movement or duplication. This integration ensures better collaboration, data reliability, and faster decision-making across the organisation.
Microsoft will introduce a new product (only private preview for now) called Data Activator. Microsoft Fabric takes data monitoring to the next level. Data Activator has the capability to monitor real-time data from multiple sources, including data lakes, Power BI, and Synapse. When Data Activator detects specific patterns, it can be programmed to initiate actions such as sending notifications or triggering business workflows. For instance, in a logistics company, Data Activator can monitor real-time events from IT devices to track package delivery statuses and send alerts in case of package overheating or driver fatigue. Its purpose is to empower business analysts and developers, putting more control in their hands.
AI Powered
The project also involves integrating new AI functionality, such as GitHub’s co-pilot and Azure’s OpenAI models like Chat GPT, to improve user productivity and enable new data interactions. Using a built-in AI assistant, data scientists can engage in a natural language dialogue to outline the steps of building a predictive churn model and generate Python code automatically.
Key Takeaways of Microsoft Fabric
UNIFIED CUSTOMER EXPERIENCE
All Microsoft’s analytics tools such as Azure Data Factory, Azure Synapse, Power BI, and Azure are integrated into Microsoft Fabric, providing a seamless experience across products, much like how Office integrates Word, PowerPoint, Excel, etc.
COST SAVINGS BY SHARING COMPUTE
Universal computing allows customers to purchase a single pool of computing power that can be used for multiple different analytics tasks. This could mean significant savings for customers.
PERSONA-CENTRIC EXPERIENCES
The new platform provides custom experiences for different roles like data engineers, data scientists, and data warehousing professionals, while ensuring all tools are integrated and feel like one product.
SHARED WORKSPACES
All artifacts live in the same workspace, making collaboration among developers easier.
ONE SECURITY
One Security will ensure that permissions will be uniformly enforced across all engines, enhancing data security and simplifying the management of security permissions for organisations using the platform.
Getting Started with Microsoft Fabric
To get started with Microsoft Fabric, you can get a free trial at
https://app.fabric.microsoft.com/home
You can activate the trial:
Once you have activated the trial. You will get 60 days to test it!
At the bottom left corner, all the key components will be available. Click on Microsoft Fabric option.
You will be redirected to the Microsoft Fabric homepage.
You are ready to start taking advantage of all the components. For example, create pipelines with Data Factory.
Conclusion
Microsoft Fabric represents a significant milestone in Microsoft’s journey to revolutionise the analytics landscape. By unifying and streamlining analytics workflows, offering a lake-centric architecture, and enabling seamless integration with Power BI and Office, Microsoft is empowering organisations to extract maximum value from their data.
Overall, Microsoft Fabric seeks to provide a unified and efficient analytics platform that simplifies the analytics process, promotes collaboration, and enhances data accessibility while prioritising security and governance.
More information at:
Many of the pictures in this blog post have been copied from Microsoft Learn and Learning Paths.
