Data factory data bricks
What is Fabric?
Fabric refers to Microsoft's unified analytics platform designed for seamless integration across various analytics services including Azure Data Factory, Azure Synapse Analytics, and others. It aims to simplify workflows by providing an end-to-end solution for managing analytics processes.
OneLake Data Hub?
OneLake Data Hub is the centralized repository, in fact, designed for vast amounts of structured and unstructured data, spread across a variety of services, within a single, cohesive ecosystem that offers easier access and integration between multiple analytics tools and provides robust governance features.
Difference between the SUM and SUMX functions in DAX?
— Answer: The SUM function in DAX is an aggregation function that calculates the sum of a numeric expression for a specified column or table
Total Sales = SUM(‘Sales’[SalesAmount])
The SUMX function, on the other hand, is an iterator function that applies a calculation expression to each row of a table and then sums up the results.
Total Sales = SUMX(‘Sales’, ‘Sales’[Quantity] * ‘Sales’[Price])
Summarize Table = SUMMARIZE(Orders, Orders[CustomerID], Orders[ProductID], “TotalSales”, SUM(Orders[Sales]))
Add column Table = ADDCOLUMNS(Orders, “Profit”, Orders[Sales] — Orders[Cost])
Category Sales = CALCULATE(SUM(‘Orders’[Sales]), FILTER(‘Orders’, ‘Orders’[Category] = “Bike”))
Data bricks -----------------------------------------------------
Databricks is a unified, cloud-based data and AI platform. Founded by the creators of Apache Spark, it provides a collaborative workspace designed to simplify and scale big data analytics, data engineering, machine learning, and generative AI workflows
• Data Lakehouse Architecture: Combines the flexible, scalable storage of a data lake with the structure and management features of a data warehouse.
• Apache Spark: Functions as a managed, high-performance computing engine, removing the complexities of setting up and scaling servers manually.
• Data Intelligence: Incorporates generative AI and natural language processing to allow users to write code, search for data, and troubleshoot using plain English
Pipeline
----------- Architect de solutions BI
ETL (transformation avant chargement)
ELT : chargement dans le Data Warehouse/Lakehouse avant transformation
La gouvernance des données désigne l'ensemble des règles, politiques et processus qui permettent à une organisation de gérer, sécuriser et valoriser ses données tout au long de leur cycle de vie. [1, 2]
Les 4 piliers fondamentaux
Qualité : Garantir que les données sont fiables, exactes et à jour.
Sécurité et confidentialité : Protéger les données contre les accès non autorisés et les pertes.
Conformité : Respecter les lois en vigueur, comme le RGPD ou les normes ISO.
Disponibilité : S'assurer que les collaborateurs autorisés peuvent accéder aux bonnes données au bon momen
Microsoft Fabric is an end-to-end, Software as a Service (SaaS) unified data analytics platform. It seamlessly combines data engineering, data integration, warehousing, data science, and business intelligence into a single, collaborative workspace. [1, 2, 3]
Core Components
• OneLake: Often described as the "OneDrive for data," it serves as a single, unified, logical data lake for your entire organization, eliminating the need to move data between different tools.
• Data Factory: Provides robust capabilities for complex data ingestion, preparation, and orchestration.
• Data Engineering & Science: Utilizes Apache Spark-powered notebooks and engines to build, train, and deploy machine learning models.
• Data Warehousing: Delivers high-performance, schema-based data warehousing within a unified ecosystem.
• Real-Time Intelligence: Simplifies working with streaming data from IoT devices and applications.
• Power BI: Deeply integrated into the platform to allow users to visualize insights and build interactive reports seamlessly on top of OneLake
•
La conception d'un entrepôt de données (ou Data Warehouse) est le processus de planification et de structuration d'un système centralisé permettant de regrouper, d'harmoniser et d'analyser de grands volumes de données issues de sources multiples pour l'aide à la decision.
Date modeling :
. Conceptual Data Modeling: A high-level, business-oriented view identifying key entities and relationships, often termed domain models.
. Logical Data Modeling: Maps business rules to technical structures (tables, columns, keys) without specific database technology details.
. Physical Data Modeling: Defines specific database implementations, including column types, indexes, and partitions, tailored to a chosen DBMS (e.g., PostgreSQL, Snowflake).. . Dimensional Modeling (Kimball): Optimized for data warehousing and analytics using star schemas (fact and dimension tables) to enable fast querying and reporting.
Benefits of BI Solution Architecture
Enhanced Decision-Making: Provides accurate, real-time insights for informed decisions.
Operational Efficiency: Automates data processes, reducing manual errors.
Scalability: Adapts to growing data volumes and evolving business needs.
Data Governance: Ensures data accuracy, security, and compliance.