Google BigQuery - gettectonic.com
Cortex Framework Integration with Salesforce (SFDC)
30Sep

Cortex Framework Integration with Salesforce (SFDC)

Cortex Framework: Integration with Salesforce (SFDC) This insight outlines the process of integrating Salesforce (SFDC) operational workloads into the Cortex Framework Data Foundation. By integrating Salesforce data through Dataflow pipelines into BigQuery, Cloud Composer can schedule and monitor these pipelines, allowing you to gain insights from your Salesforce data. Cortex Framework Integration with Salesforce explained. Prerequisite: Before configuring any workload integration, ensure that the Cortex Framework Data Foundation is deployed. Configuration File The config.json file in the Cortex Framework Data Foundation repository manages settings for transferring data from various sources, including Salesforce. Below is an example of how Salesforce workloads are configured: jsonCopy code”SFDC”: { “deployCDC”: true, “createMappingViews”: true, “createPlaceholders”: true, “datasets”: { “cdc”: “”, “raw”: “”, “reporting”: “REPORTING_SFDC” } } Explanation of Parameters: Parameter Meaning Default Value Description SFDC.deployCDC Deploy CDC true Generates Change Data Capture (CDC) processing scripts to run as DAGs in Cloud Composer. SFDC.createMappingViews Create mapping views true Creates views in the CDC processed dataset to show the “latest version of the truth” from the raw dataset. SFDC.createPlaceholders Create placeholders true Creates empty placeholder tables if they aren’t generated during ingestion, ensuring smooth downstream reporting deployment. SFDC.datasets.raw Raw landing dataset (user-defined) The dataset where replication tools land data from Salesforce. SFDC.datasets.cdc CDC processed dataset (user-defined) Source for reporting views and target for records processed by DAGs. SFDC.datasets.reporting Reporting dataset for SFDC “REPORTING_SFDC” Name of the dataset accessible for end-user reporting, where views and user-facing tables are deployed. Salesforce Data Requirements Table Structure: Loading SFDC Data into BigQuery The Cortex Framework offers several methods for loading Salesforce data into BigQuery: CDC Processing The CDC scripts rely on two key fields: You can adjust the CDC processing to handle different field names or add custom fields to suit your data schema. Configuration of API Integration and CDC To configure Salesforce data integration into BigQuery, Cortex provides the following methods: Example Configuration (settings.yaml): yamlCopy codesalesforce_to_raw_tables: – base_table: accounts raw_table: Accounts api_name: Account load_frequency: “@daily” Data Mapping and Polymorphic Fields Cortex Framework supports mapping data fields to the expected format. For example, a field named unicornId in your source system would be mapped to AccountId in Cortex with the string data type. Polymorphic Fields: Fields whose names vary but have the same structure can be mapped in Cortex using [Field Name]_Type, such as Who_Type for the Who.Type field in the Task object. Modifying DAG Templates You can customize DAG templates as needed for CDC or raw data processing. To disable CDC or raw data processing from API calls, set deployCDC=false in the configuration file. Setting Up the Extraction Module Follow these steps to set up the Salesforce to BigQuery extraction module: Cloud Composer Setup To run Python scripts for replication, install the necessary Python packages depending on your Airflow version. For Airflow 2.x: bashCopy codegcloud composer environments update my-composer-instance –location us-central1 –update-pypi-package apache-airflow-providers-salesforce>=5.2.0 Security and Permissions Ensure Cloud Composer has access to Google Secret Manager for retrieving stored secrets, enhancing the security of sensitive data like passwords and API keys. Conclusion By following these steps, you can successfully integrate Salesforce workloads into Cortex Framework, ensuring a seamless data flow from Salesforce into BigQuery for reporting and analytics. Like Related Posts Salesforce OEM AppExchange Expanding its reach beyond CRM, Salesforce.com has launched a new service called AppExchange OEM Edition, aimed at non-CRM service providers. Read more The Salesforce Story In Marc Benioff’s own words How did salesforce.com grow from a start up in a rented apartment into the world’s Read more Salesforce Jigsaw Salesforce.com, a prominent figure in cloud computing, has finalized a deal to acquire Jigsaw, a wiki-style business contact database, for Read more Health Cloud Brings Healthcare Transformation Following swiftly after last week’s successful launch of Financial Services Cloud, Salesforce has announced the second installment in its series Read more

Read More
September 30, 2024in Data, Google, Salesforce
Salesforce Data Cloud and Zero Copy
28Sep

Salesforce Data Cloud and Zero Copy

As organizations across industries gather increasing amounts of data from diverse sources, they face the challenge of making that data actionable and deriving real-time insights. With Salesforce Data Cloud and zero copy architecture, organizations can streamline access to data and build dynamic, real-time dashboards that drive value while embedding contextual insights into everyday workflows. A session during Dreamforce 2024 with Joanna McNurlen, Principal Solution Engineer for Data Cloud at Salesforce, discussed how zero copy architecture facilitates the creation of dashboards and workflows that provide near-instant insights, enabling quick decision-making to enhance operational efficiency and competitive advantage. What is zero copy architecture?Traditionally, organizations had to replicate data from one system to another, such as copying CRM data into a data warehouse for analysis. This approach introduces latency, increases storage costs, and often results in inconsistencies between systems. Zero copy architecture eliminates the need for replication and provides a single source of truth for your data. It allows different systems to access data in its original location without duplication across platforms. Instead of using traditional extract, transform, and load (ETL) processes, systems like Salesforce Data Cloud can connect directly with external databases, such as Google Cloud BigQuery, Snowflake, Databricks, or Amazon Redshift, for real-time data access. Zero copy can also facilitate data sharing from within Salesforce to other systems. As Salesforce expands its zero copy partner network, opportunities to easily connect data from various sources will continue to grow. How does zero copy work?Zero copy employs virtual tables that act as blueprints for the data structure, enabling queries to be executed as if the data were local. Changes made in the data warehouse are instantly visible across all connected systems, ensuring users always work with the latest information. While developing dashboards, users can connect directly to the zero copy objects within Data Cloud to create visualizations and reports on top of them. Why is zero copy beneficial?Zero copy allows organizations to analyze data as it is generated, enabling faster responses, smarter decision-making, and enhanced customer experiences. This architecture reduces reliance on data transformation workflows and synchronizations within both Tableau and CRM Analytics, where organizations have historically encountered bottlenecks due to runtimes and platform limits. Various teams can benefit from the following capabilities: Unlocking real-time insights in Salesforce using zero copy architectureZero copy architecture and real-time data are transforming how organizations operate. By eliminating data duplication and providing real-time insights, the use of zero copy in Salesforce Data Cloud empowers organizations to work more efficiently, make informed decisions, and enhance customer experiences. Now is the perfect time to explore how Salesforce Data Cloud and zero copy can elevate your operations. Tectonic, a trusted Salesforce partner, can help you unlock the potential of your data and create new opportunities with the Salesforce platform. Connect with us today to get started. Like Related Posts Salesforce OEM AppExchange Expanding its reach beyond CRM, Salesforce.com has launched a new service called AppExchange OEM Edition, aimed at non-CRM service providers. Read more The Salesforce Story In Marc Benioff’s own words How did salesforce.com grow from a start up in a rented apartment into the world’s Read more Salesforce Jigsaw Salesforce.com, a prominent figure in cloud computing, has finalized a deal to acquire Jigsaw, a wiki-style business contact database, for Read more Health Cloud Brings Healthcare Transformation Following swiftly after last week’s successful launch of Financial Services Cloud, Salesforce has announced the second installment in its series Read more

Read More
September 28, 2024in Data, Google, Salesforce, Salesforce Data Cloud, Snowflake
Collaborative Business Intelligence
20Sep

Collaborative Business Intelligence

Collaborative Business Intelligence: Connecting Data and Teams In today’s data-driven world, the ability to interact with business intelligence (BI) tools is essential for making informed decisions. Collaborative business intelligence (BI), also known as social BI, allows users to engage with their organization’s data and communicate with data experts through the same platforms where they already collaborate. While self-service BI empowers users to generate insights, understanding the data’s context is critical to avoid misunderstandings that can derail decision-making. Collaborative BI integrates BI tools with collaboration platforms to bridge the gap between data analysis and communication, reducing the risks of misinterpretation. Traditional Business Intelligence Traditional BI involves the use of technology to analyze data and present insights clearly. Before BI platforms became widespread, data scientists and statisticians handled data analysis, making it challenging for non-technical professionals to digest the insights. BI evolved to automate visualizations, such as charts and dashboards, making data more accessible to business users. Previously, BI reports were typically available only to high-level executives. However, modern self-service BI tools democratize access, enabling more users—regardless of technical expertise—to create reports and visualize data, fostering better decision-making across the organization. The Emergence of Collaborative BI Collaborative BI is a growing trend, combining BI applications with collaboration tools. This approach allows users to work together synchronously or asynchronously within a shared platform, making it easier to discuss data reports in real time or leave comments for others to review. Whether it’s through Slack, Microsoft Teams, or social media apps, users can receive and discuss BI insights within their usual communication channels. This seamless integration of BI and collaboration tools offers a competitive edge, simplifying the process of sharing knowledge and clarifying data without switching between applications. Key Benefits of Collaborative Business Intelligence Leading Collaborative BI Platforms Here’s a look at some of the top collaborative BI platforms driving innovation in the market: Conclusion Collaborative BI empowers organizations by improving decision-making, democratizing data access, optimizing data quality, and ensuring data security. By integrating BI tools with collaboration platforms, businesses can streamline their operations, foster a culture of data-driven decision-making, and enhance overall efficiency. Choosing the right platform is key to maximizing the benefits of collaborative BI. Like Related Posts Salesforce OEM AppExchange Expanding its reach beyond CRM, Salesforce.com has launched a new service called AppExchange OEM Edition, aimed at non-CRM service providers. Read more The Salesforce Story In Marc Benioff’s own words How did salesforce.com grow from a start up in a rented apartment into the world’s Read more Salesforce Jigsaw Salesforce.com, a prominent figure in cloud computing, has finalized a deal to acquire Jigsaw, a wiki-style business contact database, for Read more Health Cloud Brings Healthcare Transformation Following swiftly after last week’s successful launch of Financial Services Cloud, Salesforce has announced the second installment in its series Read more

Read More
September 20, 2024in Data, Google, Salesforce, Technology
Winter 25 Salesforce Release
22Aug

Get Ready for Winter 25 Salesforce Release

Salesforce Winter 25 Release notes are here. Salesforce Overall Learn about new features and enhancements that affect your Salesforce experience overall. August 8: Get early access by signing up for a Pre-Release org Admins can sign up for a pre-release Developer Edition environment, which is full of all the Winter ’25 features to explore to your heart’s content. Developer environments are stand-alone environments where you can learn, build, and get comfortable with features and functionality. If you already had a pre-release org for Summer ’24, you can log back into that one. August 14: Review the Release Notes Search the products you use for release updates in the Release Notes section of Salesforce Help. The notes will go live August 14 and we will share the link here. Get help from the community! With each release, there are a number of blogs by community members who break it down. Check out the Release Readiness Trailblazer Community Group where you can continue to get updates, share your favorite features, and ask questions about the upcoming release. August 19: Be Release Ready with Winter ’25 features for Admins Starting on August 19th, we’ll begin publishing blog posts on the Admin Blog to help you Be Release Ready with Winter ’25 features. Get ready to dive into blog posts featuring Winter ’25 user access highlights and more! As blog posts and more release resources become available, we’ll be updating the Be Release Ready page with all the resources and information you need to get started with Winter ’25. August 29 before 5 p.m. PT: Be sure to refresh your Sandbox Once you’ve explored the pre-release org and reviewed the Release Notes for features that are important to you, it’s time to try out features related to your customizations in your sandbox. This is a great time to evaluate how specific features may be useful or impact the way your organization uses Salesforce. During each release, there is a group of sandboxes slated to remain on the non-preview instance (i.e. the current release) while there is another group of sandboxes that will upgrade to the preview instance. Use the Salesforce Sandbox Preview Guide to determine the plan for your sandbox instance(s). Use the tool where you can search by sandbox instance and then specify what you want to do with your sandbox — stay on the non-preview or move to preview. It will then instruct you to refresh your sandbox to get to the desired instance or inform you that there is no action needed because your sandbox is slated for the desired instance. Contact Tectonic today if you need assistance getting Salesforce release ready. Like Related Posts Salesforce OEM AppExchange Expanding its reach beyond CRM, Salesforce.com has launched a new service called AppExchange OEM Edition, aimed at non-CRM service providers. Read more The Salesforce Story In Marc Benioff’s own words How did salesforce.com grow from a start up in a rented apartment into the world’s Read more Salesforce Jigsaw Salesforce.com, a prominent figure in cloud computing, has finalized a deal to acquire Jigsaw, a wiki-style business contact database, for Read more Health Cloud Brings Healthcare Transformation Following swiftly after last week’s successful launch of Financial Services Cloud, Salesforce has announced the second installment in its series Read more

Read More
August 22, 2024in Snowflake
Zero ETL
13Jun

Zero ETL

What is Zero-ETL? Zero-ETL represents a transformative approach to data integration and analytics by bypassing the traditional ETL (Extract, Transform, Load) pipeline. Unlike conventional ETL processes, which involve extracting data from various sources, transforming it to fit specific formats, and then loading it into a data repository, Zero-ETL eliminates these steps. Instead, it enables direct querying and analysis of data from its original source, facilitating real-time insights without the need for intermediate data storage or extensive preprocessing. This innovative method simplifies data management, reducing latency and operational costs while enhancing the efficiency of data pipelines. As the demand for real-time analytics and the volume of data continue to grow, ZETL offers a more agile and effective solution for modern data needs. Challenges Addressed by Zero-ETL Benefits of ZETL Use Cases for ZETL In Summary ZETL transforms data management by directly querying and leveraging data in its original format, addressing many limitations of traditional ETL processes. It enhances data quality, streamlines analytics, and boosts productivity, making it a compelling choice for modern organizations facing increasing data complexity and volume. Embracing Zero-ETL can lead to more efficient data processes and faster, more actionable insights, positioning businesses for success in a data-driven world. Components of Zero-ETL ZETL involves various components and services tailored to specific analytics needs and resources: Advantages and Disadvantages of ZETL Comparison: Z-ETL vs. Traditional ETL Feature Zero-ETL Traditional ETL Data Virtualization Seamless data duplication through virtualization May face challenges with data virtualization due to discrete stages Data Quality Monitoring Automated approach may lead to quality issues Better monitoring due to discrete ETL stages Data Type Diversity Supports diverse data types with cloud-based data lakes Requires additional engineering for diverse data types Real-Time Deployment Near real-time analysis with minimal latency Batch processing limits real-time capabilities Cost and Maintenance More cost-effective with fewer components More expensive due to higher computational and engineering needs Scale Scales faster and more economically Scaling can be slow and costly Data Movement Minimal or no data movement required Requires data movement to the loading stage Comparison: Zero-ETL vs. Other Data Integration Techniques Top Zero-ETL Tools Conclusion Transitioning to Zero-ETL represents a significant advancement in data engineering. While it offers increased speed, enhanced security, and scalability, it also introduces new challenges, such as the need for updated skills and cloud dependency. Zero-ETL addresses the limitations of traditional ETL and provides a more agile, cost-effective, and efficient solution for modern data needs, reshaping the landscape of data management and analytics. Like Related Posts Salesforce OEM AppExchange Expanding its reach beyond CRM, Salesforce.com has launched a new service called AppExchange OEM Edition, aimed at non-CRM service providers. Read more The Salesforce Story In Marc Benioff’s own words How did salesforce.com grow from a start up in a rented apartment into the world’s Read more Salesforce Jigsaw Salesforce.com, a prominent figure in cloud computing, has finalized a deal to acquire Jigsaw, a wiki-style business contact database, for Read more Health Cloud Brings Healthcare Transformation Following swiftly after last week’s successful launch of Financial Services Cloud, Salesforce has announced the second installment in its series Read more

Read More
June 13, 2024in Snowflake
Bring Your Own Lake With Google BigQuery
15Sep

Bring Your Own Lake With Google BigQuery

Can BigQuery Function as a Data Lake? Why you should Bring Your Own Lake With Google BigQuery. Google BigQuery serves as a fully-managed, petabyte-scale data warehouse, utilizing Google’s infrastructure’s processing power. The combination of Google Cloud Storage and BigQuery transforms Google Cloud Platform into a scalable data lake capable of storing both structured and unstructured data. Why Embrace BigQuery’s Serverless Model? In a serverless model, processing is automatically distributed across numerous machines operating in parallel. BigQuery’s serverless model allows data engineers and database administrators to concentrate less on infrastructure and more on server provisioning and deriving insights from data. Advantages of Using BigQuery as a Data Warehouse: BigQuery is a completely serverless and cost-effective cloud data warehouse designed to work across clouds, scaling seamlessly with your data. With integrated business intelligence, machine learning, and AI features, BigQuery provides a unified data platform for storing, analyzing, and sharing insights effortlessly. The Relevance of Data Lakes: Data Lakes and Data Warehouses are complementary components of data processing and reporting infrastructure, each serving distinct purposes rather than being alternatives. Data Lakes in the Evolving Landscape: Data lakes, once immensely popular, are gradually being supplanted by more advanced storage solutions like data warehouses. Data Lake Content Formats: A data lake encompasses structured data from relational databases (rows and columns), semi-structured data (CSV, logs, XML, JSON), unstructured data (emails, documents, PDFs), and binary data (images, audio, video). Building a Data Lake on GCP: Constructing a Data Lake: Introduction to Google Big Lake: BigLake serves as a storage engine, offering a unified interface for analytics and AI engines to query multiformat, multicloud, and multimodal data securely, efficiently, and in a governed manner. It aspires to create a single-copy AI lakehouse, minimizing the need for custom data infrastructure management. Data Extraction from a Data Lake: Distinguishing BigQuery as a Data Warehouse: BigQuery stands out as a serverless and cost-effective enterprise data warehouse, functioning across clouds and seamlessly scaling with data. It incorporates built-in ML/AI and BI for scalable insights. Data Lake Implementation Time: Building a fully productive data lake involves several steps, including workflow creation, security mapping, and tool and service configuration. As a result, a comprehensive data lake implementation can take several months. Acquiring a Data Lake: One option is to buy a Data Lake through a decentralized exchange (DEX) supporting the blockchain where the Data Lake resides. Connecting a crypto wallet to a DEX and utilizing a Binance account to purchase the base currency is outlined in a guide for this purpose. Like Related Posts Guide to Creating a Working Sales Plan Creating a sales plan is a pivotal step in reaching your revenue objectives. To ensure its longevity and adaptability to Read more CRM Cloud Salesforce What is a CRM Cloud Salesforce? Salesforce Service Cloud is a customer relationship management (CRM) platform for Salesforce clients to Read more Web Pages That Helped With My Google Data Engineer Exam Google Data Engineer Exam It seems like every day more resources appear to help you study for the Google Data Read more What is Advanced Reporting in Salesforce? Cross Filters, Summary Formulas, and More: Advanced Reporting in Salesforce Salesforce comes with report types out-of-the-box for all standard objects Read more

Read More
September 15, 2023in Google Analytics, Salesforce Data Cloud

Subscribe to our mailing list. Join our mail list to receive our newsletter

We'll keep you up to date with our latest news, insights, free resources, and much more.
Don't worry, we won't spam you.
gettectonic.com