Skillquality 0.70

azure-databricks

Expert knowledge for Azure Databricks development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when using Unity Catalog, Delta/Lakeflow pip

Price
free
Protocol
skill
Verified
no

What it does

Azure Databricks Skill

This skill provides expert guidance for Azure Databricks. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g., L35-L120), use read_file with the specified lines. For categories with file links (e.g., [security.md](security.md)), use read_file on the linked reference file

IMPORTANT for Agent: If metadata.generated_at is more than 3 months old, suggest the user pull the latest version from the repository. If mcp_microsoftdocs tools are not available, suggest the user install it: Installation Guide

This skill requires network access to fetch documentation content:

  • Preferred: Use mcp_microsoftdocs:microsoft_docs_fetch with query string from=learn-agent-skill. Returns Markdown.
  • Fallback: Use fetch_webpage with query string from=learn-agent-skill&accept=text/markdown. Returns Markdown.

Category Index

CategoryLocationDescription
TroubleshootingL37-L139Diagnosing and fixing Databricks issues: cluster startup/termination, Spark and SQL errors, connectors/Lakeflow ingestion, VS Code/CLI/Connect, model serving, AI agents, and performance debugging.
Best PracticesL140-L308Best-practice guidance for Databricks architecture, performance, cost, governance, streaming, Lakeflow, ML/LLM/MLOps, Vector Search, BI, and operational reliability across the lakehouse.
Decision MakingL309-L400Guides for architectural and migration decisions: choosing compute, runtimes, Unity Catalog, budgets, networking, ML/LLM options, ingestion, and lakehouse deployment patterns.
Architecture & Design PatternsL401-L439Architectural blueprints and patterns for Databricks: lakehouse, networking, storage, HA/DR, governance, performance, cost, streaming, Lakebase, ML/MLOps, RAG, agents, and IDP pipelines.
Limits & Quotaslimits-quotas.mdQuotas, limits, and constraints for Azure Databricks compute, connectors, Lakeflow, Lakebase, model serving, tokens, APIs, data types, dashboards, and Unity Catalog resources.
Securitysecurity.mdSecurity, identity, and compliance for Azure Databricks: authN/authZ, Unity Catalog/ABAC, networking, encryption, secrets, audit logs, compliance controls, and secure external/ingestion integrations.
Configurationconfiguration.mdConfiguring and administering Azure Databricks: accounts, workspaces, security, networking, compute, jobs, data/UC/Delta/Lakeflow, ML/serving, agents, Marketplace, and SQL/runtime settings.
Integrations & Coding Patternsintegrations.mdPatterns and APIs for integrating Databricks with external data systems, tools, and AI/ML frameworks, plus detailed PySpark/SQL function references and Lakehouse Federation connectivity.
Deploymentdeployment.mdDeploying and operating Databricks workspaces, CI/CD, IaC, model/feature serving, AI agents, dashboards, and migrations (Unity Catalog, Feature Store, routing, charts) across Azure environments

Troubleshooting

TopicURL
Interpret Azure Databricks diagnostic audit log eventshttps://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/audit-logs
Troubleshoot Azure Databricks compute startup issueshttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/
Resolve Databricks classic compute termination error codeshttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/cluster-error-codes
Debug Spark applications using Databricks Spark UIhttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/debugging-spark-ui
Troubleshoot Apache Kafka usage on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/connect/streaming/kafka/faq
Diagnose and fix common Delta Sharing errorshttps://learn.microsoft.com/en-us/azure/databricks/delta-sharing/troubleshooting
Troubleshoot common Databricks CLI issueshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/troubleshooting
Diagnose and fix Databricks Connect Python issueshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/troubleshooting
Diagnose and fix Databricks Connect Scala issueshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/troubleshooting
Troubleshoot common Databricks Terraform provider errorshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/terraform/troubleshoot
Resolve common issues with Databricks VS Code extensionhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/faqs
Troubleshoot Databricks VS Code extension errorshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/troubleshooting
Resolve ARITHMETIC_OVERFLOW errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/arithmetic-overflow-error-class
Handle CAST_INVALID_INPUT errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/cast-invalid-input-error-class
Diagnose DC_GA4_RAW_DATA_ERROR in GA4 connectorhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-ga4-raw-data-error-error-class
Understand DC_SFDC_API_ERROR in Databricks connectorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sfdc-api-error-error-class
Diagnose DC_SQLSERVER_ERROR in SQL Server connectorhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sqlserver-error-error-class
Understand DELTA_ICEBERG_COMPAT_V1_VIOLATION errorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/delta-iceberg-compat-v1-violation-error-class
Handle DIVIDE_BY_ZERO errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/divide-by-zero-error-class
Handle Azure Databricks named error conditionshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/error-classes
Fix EWKB_PARSE_ERROR geometry parsing issueshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkb-parse-error-error-class
Fix EWKT_PARSE_ERROR geometry parsing issueshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkt-parse-error-error-class
Resolve GEOJSON_PARSE_ERROR in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/geojson-parse-error-error-class
Address GROUP_BY_AGGREGATE errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/group-by-aggregate-error-class
Handle H3_INVALID_CELL_ID errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-cell-id-error-class
Interpret and resolve H3_INVALID_GRID_DISTANCE_VALUE in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-grid-distance-value-error-class
Handle H3_INVALID_RESOLUTION_VALUE errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-resolution-value-error-class
Resolve H3_NOT_ENABLED errors and tier requirementshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-not-enabled-error-class
Fix INSUFFICIENT_TABLE_PROPERTY errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/insufficient-table-property-error-class
Troubleshoot INVALID_ARRAY_INDEX errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-error-class
Troubleshoot INVALID_ARRAY_INDEX_IN_ELEMENT_AT in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-in-element-at-error-class
Resolve MISSING_AGGREGATION errors in Databricks querieshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/missing-aggregation-error-class
Diagnose ROW_COLUMN_ACCESS errors for filters and maskshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/row-column-access-error-class
Interpret Azure Databricks SQLSTATE error codeshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/sqlstates
Fix TABLE_OR_VIEW_NOT_FOUND errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/table-or-view-not-found-error-class
Resolve UNRESOLVED_ROUTINE function resolution errorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/unresolved-routine-error-class
Understand UNSUPPORTED_TABLE_OPERATION errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-table-operation-error-class
Understand UNSUPPORTED_VIEW_OPERATION errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-view-operation-error-class
Troubleshoot WKB_PARSE_ERROR for geometry parsinghttps://learn.microsoft.com/en-us/azure/databricks/error-messages/wkb-parse-error-error-class
Troubleshoot WKT_PARSE_ERROR for geometry parsinghttps://learn.microsoft.com/en-us/azure/databricks/error-messages/wkt-parse-error-error-class
Troubleshoot MLflow 2 Agent Evaluation issueshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/troubleshooting
Troubleshoot and debug Databricks AI agentshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/debug-agent
Diagnose and fix common Genie Space issueshttps://learn.microsoft.com/en-us/azure/databricks/genie/troubleshooting
Troubleshoot common Databricks Auto Loader issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/faq
Resolve common Confluence connector ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/confluence-faq
Troubleshoot authentication and rate limit errors for Confluencehttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/confluence-troubleshoot
Troubleshoot Dynamics 365 Lakeflow ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/d365-troubleshoot
Resolve common issues with Lakeflow managed connectorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/faq
Troubleshoot Google Ads connector ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-ads-troubleshoot
Troubleshoot Google Analytics raw data ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-analytics-troubleshoot
Troubleshoot common HubSpot connector ingestion problemshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/hubspot-troubleshoot
Troubleshoot Jira Lakeflow ingestion errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/jira-troubleshoot
Troubleshoot Meta Ads ingestion connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/meta-ads-troubleshoot
Diagnose and fix MySQL Lakeflow Connect ingestionhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/mysql-troubleshoot
Troubleshoot common Outlook connector ingestion errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/outlook-troubleshoot
Troubleshoot PostgreSQL Lakeflow Connect ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-troubleshoot
Troubleshoot Lakeflow Connect query-based connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/query-based-troubleshoot
Troubleshoot Salesforce Lakeflow ingestion problemshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-troubleshoot
Diagnose and fix Databricks ServiceNow connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/servicenow-troubleshoot
Diagnose and fix Lakeflow SharePoint connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sharepoint-troubleshoot
Troubleshoot Databricks Smartsheet connector errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/smartsheet-troubleshoot
Answer common SQL Server Lakeflow Connect connector questionshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-faq
Resolve SQL Server Lakeflow Connect ingestion problemshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-troubleshoot
Troubleshoot TikTok Ads connector in Lakeflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/tiktok-ads-troubleshoot
Troubleshoot Workday HCM connector in Lakeflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-hcm-troubleshoot
Diagnose and fix Databricks Workday connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-reports-troubleshoot
Resolve common Zendesk Support connector errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/zendesk-support-troubleshoot
Handle Zerobus Ingest errors and retrieshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-errors
Inspect logs for Databricks init script executionhttps://learn.microsoft.com/en-us/azure/databricks/init-scripts/logs
Test and validate Databricks ODBC driver connectionshttps://learn.microsoft.com/en-us/azure/databricks/integrations/odbc/testing
Configure and troubleshoot Lakeflow Jobs with many taskshttps://learn.microsoft.com/en-us/azure/databricks/jobs/large-jobs
Monitor and inspect Lakeflow Jobs runs and historyhttps://learn.microsoft.com/en-us/azure/databricks/jobs/monitor
Troubleshoot and repair Azure Databricks Lakeflow job failureshttps://learn.microsoft.com/en-us/azure/databricks/jobs/repair-job-failures
Monitor and troubleshoot Lakeflow Spark pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/observability
Use pipeline query history for debugging and tuninghttps://learn.microsoft.com/en-us/azure/databricks/ldp/query-history
Recover Lakeflow pipelines from streaming checkpoint failureshttps://learn.microsoft.com/en-us/azure/databricks/ldp/recover-streaming
User guides, migration, and troubleshooting for AI Runtimehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ai-runtime/guides
Diagnose and resolve Databricks Feature Store issueshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/troubleshooting-and-limitations
Debug common Databricks model serving issueshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-debug
Diagnose Databricks model serving issues with Genie Codehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-genie-code
Debug Python code in Databricks notebookshttps://learn.microsoft.com/en-us/azure/databricks/notebooks/debugger
Troubleshoot failing Spark jobs and executors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/failing-spark-jobs
Use Databricks Spark jobs timeline for debugginghttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/jobs-timeline
Diagnose long-running Spark stages in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage
Investigate high I/O Spark stages in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage-io
Debug slow low-I/O Spark stages in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/slow-spark-stage-low-io
Identify expensive reads in Spark DAG on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-dag-expensive-read
Diagnose gaps between Spark jobs in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-job-gaps
Diagnose and fix Spark memory issues on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-memory-issues
Troubleshoot Azure Databricks Partner Connect issueshttps://learn.microsoft.com/en-us/azure/databricks/partner-connect/troubleshoot
Troubleshoot common Azure Databricks Git folder errorshttps://learn.microsoft.com/en-us/azure/databricks/repos/errors-troubleshooting
Fetch cursor rows and handle SQLSTATE in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/fetch-stmt
Use GET DIAGNOSTICS for SQL error handling in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/get-diagnostics-stmt
Open cursors and handle errors with OPEN in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/open-stmt
Validate UTF-8 strings and handle INVALID_UTF8_STRINGhttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/validate_utf8
Interpret Databricks SQL query performance insightshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/performance-insights
Use Databricks SQL query history to debug performancehttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-history
Analyze Databricks SQL query profiles to find bottleneckshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-profile
Troubleshoot and configure Databricks SQL scheduled querieshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/schedule-query

Best Practices

TopicURL
Apply Databricks usage tags for accurate cost attributionhttps://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage-detail-tags
Use default Databricks policy families to enforce compute best practiceshttps://learn.microsoft.com/en-us/azure/databricks/admin/clusters/policy-families
Apply Azure Databricks identity configuration best practiceshttps://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/best-practices
Configure default deletion vectors for Databricks Delta tableshttps://learn.microsoft.com/en-us/azure/databricks/admin/workspace-settings/deletion-vectors
Apply best practices for Azure Databricks serverless workspaceshttps://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces-best-practices
Migrate Databricks library installs from init scriptshttps://learn.microsoft.com/en-us/azure/databricks/archive/compute/libraries-init-scripts
Apply compute policy best practices in Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/archive/compute/policies-best-practices
Use DBIO for transactional writes to cloud storage in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/archive/legacy/dbio-commit
Optimize skewed joins in Databricks using skew hintshttps://learn.microsoft.com/en-us/azure/databricks/archive/legacy/skew-join
Migrate from Databricks Deep Learning Pipelineshttps://learn.microsoft.com/en-us/azure/databricks/archive/spark-3.x-migration/deep-learning-pipelines
Use advanced techniques in Databricks metric viewshttps://learn.microsoft.com/en-us/azure/databricks/business-semantics/metric-views/advanced-techniques
Apply Azure Databricks administration best practiceshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/administration
Optimize BI performance with Databricks SQL warehouseshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving
Prepare and model data for high-performance BI on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-data-prep
Configure Databricks SQL warehouses for optimal BI servinghttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-sql-serving
Apply Azure Databricks compute creation best practiceshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/compute
Implement Azure Databricks production job scheduling best practiceshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/jobs
Best practices for Power BI dashboards on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/power-bi
Apply Databricks compute configuration best practiceshttps://learn.microsoft.com/en-us/azure/databricks/compute/cluster-config-best-practices
Use flexible node types for reliable Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/flexible-node-types
Apply best practices for Databricks poolshttps://learn.microsoft.com/en-us/azure/databricks/compute/pool-best-practices
Apply serverless compute best practices in Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/compute/serverless/best-practices
Tune Databricks SQL warehouses for BI workloadshttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/bi-workload-settings
Use system table queries to monitor Databricks SQL warehouseshttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/monitor/queries
Control large interactive queries with Query Watchdoghttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/query-watchdog
Apply observability best practices for Databricks jobs and pipelineshttps://learn.microsoft.com/en-us/azure/databricks/data-engineering/observability-best-practices
Best practices for designing Unity Catalog ABAC policieshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/best-practices
Optimize performance of Unity Catalog ABAC policieshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/performance
Apply Unity Catalog data governance best practiceshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/best-practices
Apply row filters and column masks in Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/filters-and-masks/
Work with legacy Hive metastore database objectshttps://learn.microsoft.com/en-us/azure/databricks/database-objects/hive-metastore
Follow DBFS root storage recommendations in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/dbfs/dbfs-root
Migrate from DBFS mounts to Unity Catalog external locationshttps://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts
Apply DBFS and Unity Catalog usage best practiceshttps://learn.microsoft.com/en-us/azure/databricks/dbfs/unity-catalog
Optimize Delta Sharing to reduce cloud egress costshttps://learn.microsoft.com/en-us/azure/databricks/delta-sharing/manage-egress
Apply Delta Lake best practices on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/best-practices
Optimize tables with liquid clustering instead of partitioninghttps://learn.microsoft.com/en-us/azure/databricks/delta/clustering
Tune Azure Databricks data skipping with stats and Z-orderhttps://learn.microsoft.com/en-us/azure/databricks/delta/data-skipping
Use deletion vectors to optimize Delta table updateshttps://learn.microsoft.com/en-us/azure/databricks/delta/deletion-vectors
Drop or replace Delta and Unity Catalog tables safelyhttps://learn.microsoft.com/en-us/azure/databricks/delta/drop-table
Optimize Delta table file layout on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/optimize
Handle Delta Lake limitations on Amazon S3https://learn.microsoft.com/en-us/azure/databricks/delta/s3-limitations
Use selective overwrite options in Delta Lakehttps://learn.microsoft.com/en-us/azure/databricks/delta/selective-overwrite
Control Delta table data file size on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/tune-file-size
Vacuum Delta tables and manage retention safelyhttps://learn.microsoft.com/en-us/azure/databricks/delta/vacuum
Optimize VARIANT data performance with shredding on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/variant-shredding
Apply MLOps Stack best practices with bundleshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/mlops-stacks
Apply CI/CD best practices on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/best-practices
View Databricks cluster policy families via CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/policy-families-commands
Apply security and performance best practices for Databricks appshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/best-practices
Test Databricks Connect for Python code with pytesthttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/testing
Handle async queries and interruptions in Databricks Connecthttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/queries
Choose between Databricks volumes and workspace fileshttps://learn.microsoft.com/en-us/azure/databricks/files/files-recommendations
Orchestrate multi-agent systems with Supervisor Agenthttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-bricks/multi-agent-supervisor
Apply best practices for MLflow 2 evaluation setshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/evaluation-set
Follow an end-to-end Databricks agents development workflowhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/guide/agents-dev-workflow
Measure RAG performance with Databricks metricshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/evaluate-assess-performance
Evaluate and monitor RAG apps on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/fundamentals-evaluation-monitoring-rag
Optimize Databricks RAG application qualityhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-overview
Improve Databricks RAG chain qualityhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain
Write effective custom instructions for Genie Codehttps://learn.microsoft.com/en-us/azure/databricks/genie-code/instructions
Apply practical tips to improve Genie Code responseshttps://learn.microsoft.com/en-us/azure/databricks/genie-code/tips
Use Genie Agent mode for complex analysishttps://learn.microsoft.com/en-us/azure/databricks/genie/agent-mode
Evaluate Genie Spaces using benchmarkshttps://learn.microsoft.com/en-us/azure/databricks/genie/benchmarks
Apply best practices to curate Genie Spaceshttps://learn.microsoft.com/en-us/azure/databricks/genie/best-practices
Migrate existing Auto Loader streams to file eventshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/migrating-to-file-events
Configure Auto Loader for production workloadshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/production
Apply common COPY INTO data loading patternshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/copy-into/examples
Apply common patterns for Lakeflow ingestionhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/common-patterns
Perform full refreshes of Lakeflow target tableshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/full-refresh
Maintain Lakeflow managed ingestion pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/pipeline-maintenance
Maintain and operate PostgreSQL ingestion pipelines in Lakeflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-maintenance
Enable incremental ingestion for Salesforce formula fieldshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-formula-fields
Use Databricks init scripts for cluster customizationhttps://learn.microsoft.com/en-us/azure/databricks/init-scripts/
Reference external files safely in Databricks init scriptshttps://learn.microsoft.com/en-us/azure/databricks/init-scripts/referencing-files
Configure and optimize compute for Lakeflow Jobshttps://learn.microsoft.com/en-us/azure/databricks/jobs/compute
Build metadata-driven For each jobs with control tableshttps://learn.microsoft.com/en-us/azure/databricks/jobs/how-to/foreach-sql-lookup-tutorial
Apply best practices for configuring classic Lakeflow Jobshttps://learn.microsoft.com/en-us/azure/databricks/jobs/run-classic-jobs
Apply cost optimization best practices on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/cost-optimization/best-practices
Implement best practices for Databricks data and AI governancehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/best-practices
Design observability and monitoring strategy for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/observability
Apply interoperability and usability best practices on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/interoperability-and-usability/best-practices
Implement operational excellence best practices on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/operational-excellence/best-practices
Apply performance best practices for Azure Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/performance-efficiency/best-practices
Apply reliability best practices on Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reliability/best-practices
Implement security and compliance best practices in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/security-compliance-and-privacy/best-practices
Optimize Lakeflow pipelines with enhanced autoscalinghttps://learn.microsoft.com/en-us/azure/databricks/ldp/auto-scaling
Apply best practices for Lakeflow Spark Declarative Pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/best-practices
Use advanced AUTO CDC features and monitor processing metricshttps://learn.microsoft.com/en-us/azure/databricks/ldp/cdc-advanced
Develop and test Lakeflow Spark Declarative Pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/develop
Manage Python dependencies safely in Databricks pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/developer/external-dependencies
Implement advanced expectation patterns at scalehttps://learn.microsoft.com/en-us/azure/databricks/ldp/expectation-patterns
Reduce high initialization times in Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/fix-high-init
Backfill historical data with Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/flows-backfill
Run full refresh operations for Databricks streaming tables safelyhttps://learn.microsoft.com/en-us/azure/databricks/ldp/full-refresh-st
Optimize stateful stream processing with watermarkshttps://learn.microsoft.com/en-us/azure/databricks/ldp/stateful-processing
Design CDC and snapshot patterns in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/ldp/what-is-change-data-capture
Restart the Python process to refresh Databricks librarieshttps://learn.microsoft.com/en-us/azure/databricks/libraries/restart-python-process
Apply data loading best practices on Databricks AI Runtimehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ai-runtime/dataloading
Apply Hyperopt best practices on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/automl-hyperparam-tuning/hyperopt-best-practices
Improve Databricks AutoML forecasting with covariateshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/automl/automl-covariate-forecast
Implement point-in-time correct feature joins for time series MLhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/time-series
Benchmark Databricks LLM endpoints for latency and TPShttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/foundation-model-apis/prov-throughput-run-benchmark
Implement LLMOps workflows on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/llmops
Validate models before Databricks serving deploymenthttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-pre-deployment-validation
Monitor Databricks model quality and endpoint healthhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/monitor-diagnose-endpoints
Optimize Databricks Model Serving endpoints for productionhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/production-optimization
Load test Databricks Mosaic AI serving endpointshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/what-is-load-test
Tune and scale Ray clusters on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ray/scale-ray
Follow deep learning best practices on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/dl-best-practices
Fine-tune Hugging Face models on a single GPU in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/huggingface/fine-tune-model
Prepare datasets for Hugging Face fine-tuning on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/huggingface/load-data
Adapt Apache Spark workloads for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/migration/spark
Align MLflow LLM judges with human evaluatorshttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/align-judges
Evaluate and compare MLflow prompt versions effectivelyhttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/prompt-version-mgmt/prompt-registry/evaluate-prompts
Use manual MLflow tracing for production GenAI appshttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/tracing/app-instrumentation/manual-tracing/
Analyze GenAI traces for errors and performancehttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/tracing/observe-with-traces/analyze-traces
Run Databricks notebooks safely and efficientlyhttps://learn.microsoft.com/en-us/azure/databricks/notebooks/run-notebook
Test and schedule Databricks notebook codehttps://learn.microsoft.com/en-us/azure/databricks/notebooks/test-notebooks
Create and manage Lakebase read replicashttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/manage-read-replicas
Monitor Lakebase queries using pg_stat_statementshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/pg-stat-statements
Apply performance optimization recommendations on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/
Use adaptive query execution on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/aqe
Leverage cost-based optimizer in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/cbo
Improve read performance with Databricks disk cachehttps://learn.microsoft.com/en-us/azure/databricks/optimizations/disk-cache
Improve Delta query performance with dynamic file pruning on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/dynamic-file-pruning
Accelerate data access with predictive I/Ohttps://learn.microsoft.com/en-us/azure/databricks/optimizations/predictive-io
Use predictive optimization for Unity Catalog tableshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/predictive-optimization
Tune Azure Databricks range join optimizationhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/range-join
Diagnose Databricks Spark cost and performance in UIhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/
Debug skew and spill in Databricks Spark stageshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage-page
Handle Databricks spot instance losses effectivelyhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/losing-spot-instances
Resolve long Spark stages with a single taskhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/one-spark-task
Optimize many small Spark jobs on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/small-spark-jobs
Mitigate overloaded Spark driver on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-driver-overloaded
Detect unnecessary data rewriting in Databricks Spark writeshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-rewriting-data
Best practices for setting up Databricks Partner Connecthttps://learn.microsoft.com/en-us/azure/databricks/partner-connect/best-practice
Optimize joins with broadcast hints in Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/pyspark/reference/functions/broadcast
Configure networking for Databricks Lakehouse Federation data sourceshttps://learn.microsoft.com/en-us/azure/databricks/query-federation/networking
Optimize performance of Databricks Lakehouse Federation querieshttps://learn.microsoft.com/en-us/azure/databricks/query-federation/performance-recommendations
Transform complex and nested data types in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/complex-types
Use higher-order functions on arrays in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/higher-order-functions
Differences between VARIANT and JSON strings in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/variant-json-diff
Work with OBJECT type and VARIANT schemas in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/object-type
Use TIMESTAMP_NTZ type and Delta support in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/timestamp-ntz-type
Use VARIANT type and Iceberg compatibility in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/variant-type
Collect table statistics with ANALYZE TABLE for optimizationhttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-aux-analyze-compute-statistics
Benchmark Databricks SQL warehouses with the TPC-DS datasethttps://learn.microsoft.com/en-us/azure/databricks/sql/tpcds-eval
Optimize Databricks SQL queries using primary key constraintshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-optimization-constraints
Use Structured Streaming checkpoints safely on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/checkpoints
Configure Databricks Structured Streaming for productionhttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/production
Optimize and monitor Databricks real-time streaming performancehttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/real-time/performance
Optimize stateless Structured Streaming queries on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/stateless-streaming
Apply watermarks for stateful streaming on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/watermarks
Use automatic Unity Catalog table upgrades in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/tables/automatic-upgrades
Analyze Databricks table size and optimize storage costshttps://learn.microsoft.com/en-us/azure/databricks/tables/size
Design data models optimized for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/transform/data-modeling
Optimize join performance for Azure Databricks workloadshttps://learn.microsoft.com/en-us/azure/databricks/transform/optimize-joins
Clean and validate data with Databricks batch and streaminghttps://learn.microsoft.com/en-us/azure/databricks/transform/validate
Optimize Unity Catalog batch Python UDF performancehttps://learn.microsoft.com/en-us/azure/databricks/udf/python-batch-udf
Tune Mosaic AI Vector Search performance and latencyhttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-best-practices
Design and run load tests for Vector Search endpointshttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-endpoint-load-test
Improve Mosaic AI Vector Search retrieval qualityhttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-retrieval-quality
Identify and clean up unused Databricks Vector Search endpointshttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-unused-endpoints
Download internet data into Azure Databricks volumeshttps://learn.microsoft.com/en-us/azure/databricks/volumes/download-internet-files

Decision Making

TopicURL
Manage and change Azure Databricks subscription tierhttps://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/account
Create and manage Databricks budgets to track usagehttps://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/budgets
Plan migration from Standard to Premium Databricks workspaceshttps://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/standard-tier
Decide when and how to use serverless Databricks workspaceshttps://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces
Decide and migrate from dbx to Databricks bundleshttps://learn.microsoft.com/en-us/azure/databricks/archive/dev-tools/dbx/dbx-migrate
Migrate optimized LLM endpoints to provisioned throughputhttps://learn.microsoft.com/en-us/azure/databricks/archive/machine-learning/migrate-provisioned-throughput
Decide when to use Databricks Light runtimehttps://learn.microsoft.com/en-us/azure/databricks/archive/runtime/light
Plan migration of Databricks workloads to Spark 3.xhttps://learn.microsoft.com/en-us/azure/databricks/archive/spark-3.x-migration/
Select and manage the default Unity Catalog cataloghttps://learn.microsoft.com/en-us/azure/databricks/catalogs/default
Choose appropriate Azure Databricks compute typeshttps://learn.microsoft.com/en-us/azure/databricks/compute/choose-compute
Select compatible flexible node types for Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/flexible-node-type-instances
Decide when and how to use GPU Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/gpu
Decide when and how to use Azure Databricks poolshttps://learn.microsoft.com/en-us/azure/databricks/compute/pool-index
Plan migration from classic to serverless Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/serverless/migration
Choose and manage Azure Databricks SQL warehouse sizing and scalinghttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/warehouse-behavior
Choose between Databricks SQL warehouse typeshttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/warehouse-types
Choose between ABAC and table-level filters in Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/abac-vs-rls-cm
Choose between managed and external Unity Catalog assetshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/managed-versus-external
Plan and execute upgrade of Databricks workspaces to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/upgrade/
Prepare and migrate to Unity Catalog–only Databricks workspaceshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/upgrade/uc-only-migration
Choose Delta Lake protocol versions and feature setshttps://learn.microsoft.com/en-us/azure/databricks/delta/feature-compatibility
Choose local development tools for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/
Migrate from legacy to new Databricks CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/migrate
Manage Databricks account budget policies via CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/account-budget-policy-commands
Configure Databricks account budgets using CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/account-budgets-commands
Manage Databricks account usage dashboards via CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/account-usage-dashboards-commands
Plan migration from legacy Databricks Connect runtimeshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect-legacy
Migrate from older to new Databricks Connect for Pythonhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/migrate
Migrate from legacy to new Scala Databricks Connecthttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/migrate
Choose and use Databricks SDKs for automationhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/sdks
Decide between CDKTF and Databricks Terraform providerhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/terraform/cdktf
Choose Unity Catalog integration method for external engineshttps://learn.microsoft.com/en-us/azure/databricks/external-access/integrations
Decide when to migrate agents to Databricks Appshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/migrate-agent-to-apps
Use external models with Mosaic AI Model Servinghttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/external-models/
Select Azure Databricks generative AI capabilities for your workflowhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/guide/gen-ai-capabilities
Choose between Databricks Free Edition and free trialhttps://learn.microsoft.com/en-us/azure/databricks/getting-started/free-trial-vs-free-edition
Choose incremental ingestion options from cloud object storagehttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/
Select Auto Loader file detection mode for your workloadhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/file-detection-modes
Plan migration of existing data to Delta Lake on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/data-migration/
Choose and set up MySQL Lakeflow ingestionhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/mysql
Plan and configure PostgreSQL Lakeflow ingestionhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql
Plan Microsoft SQL Server Lakeflow ingestion workflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-overview
Choose and start with Databricks ODBC and JDBC drivershttps://learn.microsoft.com/en-us/azure/databricks/integrations/jdbc-odbc-bi
Migrate from Simba Spark ODBC to Databricks ODBChttps://learn.microsoft.com/en-us/azure/databricks/integrations/odbc/migration
Plan and manage production workloads with Lakeflow Jobshttps://learn.microsoft.com/en-us/azure/databricks/jobs/
Decide when to run Lakeflow Jobs on serverless computehttps://learn.microsoft.com/en-us/azure/databricks/jobs/run-serverless-jobs
Migrate from Spark Submit tasks in Databricks jobshttps://learn.microsoft.com/en-us/azure/databricks/jobs/spark-submit
Plan production Azure Databricks lakehouse deploymentshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/
Design compute and workspace configuration for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/compute
Choose a programming language for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/languages/overview
Choose triggered vs continuous mode for pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/pipeline-mode
Upgrade workspace feature tables to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/uc/upgrade-feature-table-to-uc
Select Databricks-hosted foundation models for APIshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/foundation-model-apis/supported-models
Migrate MLflow model versions to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/migrate-models
Decide and migrate to Unity Catalog model managementhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/migrate-to-uc
Upgrade Databricks ML workflows to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/upgrade-workflows
Choose Databricks options for batch model inferencehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-inference/
Migrate from legacy to Mosaic AI Model Servinghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/migrate-model-serving
Decide when to use Spark vs. Ray on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ray/spark-ray-overview
Understand Azure Databricks generative model maintenance policyhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/retired-models-policy
Plan migration of data applications to Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/migration/
Scope and plan ETL pipeline migration to Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/migration/etl
Choose a migration path from Parquet to Delta Lakehttps://learn.microsoft.com/en-us/azure/databricks/migration/parquet-to-delta-lake
Plan migration from data warehouse to Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/migration/warehouse-to-lakehouse
Decide and migrate from Agent Evaluation to MLflow 3https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/agent-eval-migration
Quick reference for migrating to MLflow 3https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/agent-eval-migration-reference
Choose between open source and managed MLflow on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/overview/oss-managed-diff
Choose compute resources for Databricks notebookshttps://learn.microsoft.com/en-us/azure/databricks/notebooks/notebook-compute
Right-size Lakebase instance capacity and scalinghttps://learn.microsoft.com/en-us/azure/databricks/oltp/instances/create/capacity
Choose backup and restore methods for Lakebasehttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/backup-methods
Decide when and how to use Lakebase Autoscaling by defaulthttps://learn.microsoft.com/en-us/azure/databricks/oltp/upgrade-to-autoscaling
Configure incremental refresh for Databricks materialized viewshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/incremental-refresh
Choose pandas options on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/pandas/
Use Hive metastore federation in Unity Catalog migrationshttps://learn.microsoft.com/en-us/azure/databricks/query-federation/hms-federation-concepts
Migrate legacy Databricks query federation to Lakehouse Federationhttps://learn.microsoft.com/en-us/azure/databricks/query-federation/migrate
Plan and execute migration to Databricks Runtime 11.xhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/11.x-migration
Migrate workloads to Databricks Runtime 12.x safelyhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/12.x-migration
Migrate workloads to Databricks Runtime 13.x safelyhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/13.x-migration
Migrate workloads to Databricks Runtime 14.x safelyhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/14.x-migration
Plan around Databricks Runtime and feature lifecycleshttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/databricks-runtime-ver
Understand serverless DBU billing by Azure Databricks SKUhttps://learn.microsoft.com/en-us/azure/databricks/resources/pricing
Plan and manage Azure Databricks serverless networking costshttps://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/cost-management
Decide between Spark Connect and Spark Classichttps://learn.microsoft.com/en-us/azure/databricks/spark/connect-vs-classic
Choose between SparkR and sparklyr on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sparkr/sparkr-vs-sparklyr
Use SYNC to upgrade Hive tables to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-aux-sync
Choose Structured Streaming output modes on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/output-mode
Choose and implement Databricks transaction modeshttps://learn.microsoft.com/en-us/azure/databricks/transactions/transaction-modes
Optimize and manage Mosaic AI Vector Search costshttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-cost-management

Architecture & Design Patterns

TopicURL
Plan disaster recovery architecture for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/admin/disaster-recovery
Design and use materialization for Databricks metric viewshttps://learn.microsoft.com/en-us/azure/databricks/business-semantics/metric-views/materialization
Implement fan-in and fan-out in Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/data-engineering/fan-in-fan-out
Choose patterns for external access to Databricks datahttps://learn.microsoft.com/en-us/azure/databricks/external-access/
Build an IDP pipeline with Databricks AI Functionshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-bricks/idp-pipeline-tutorial
Design intelligent document processing pipelines on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-bricks/intelligent-document-processing
Design multi-agent orchestrator apps on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/multi-agent-apps
Apply agent system design patterns on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/guide/agent-system-design-patterns
Design measurement infrastructure for RAG quality on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/evaluate-enable-measurement
Design and tune Databricks RAG inference chainshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/fundamentals-inference-chain-rag
Design cost optimization architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/cost-optimization/
Apply data and AI governance architecture on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/
Design Delta Lake and medallion data architecture on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/delta-lake
Design high availability and disaster recovery for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/ha-dr
Design Azure Databricks network and connectivity architecturehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/network
Design storage architecture for Azure Databricks and Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/storage
Design Azure Databricks workspace architecture strategyhttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/workspace-strategy
Design interoperability and usability architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/interoperability-and-usability/
Design operational excellence architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/operational-excellence/
Design performance efficiency architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/performance-efficiency/
Apply Azure Databricks lakehouse reference architectureshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reference
Design reliability architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reliability/
Apply medallion lakehouse architecture on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse/medallion
Choose Databricks ML model deployment patternshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/deployment-patterns
Implement MLOps workflows on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/mlops-workflow
Choose and train deep learning recommender models on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-recommender-models
Use Lakebase branches for database development workflowshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/branches
Design for high availability with Lakebase computeshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/high-availability
Scale reads with Lakebase read replicashttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/read-replicas
Connect Databricks Serverless Private Git to on-prem Githttps://learn.microsoft.com/en-us/azure/databricks/repos/connect-on-prem-git-server
Set up Databricks Serverless Private Git with Private Linkhttps://learn.microsoft.com/en-us/azure/databricks/repos/serverless-private-git
Choose patterns for modeling semi-structured data on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/
Choose async checkpointing for Databricks stateful querieshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/async-checkpointing
Use async progress tracking in Databricks streaminghttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/async-progress-checking
Decide when and how to partition Delta tableshttps://learn.microsoft.com/en-us/azure/databricks/tables/partitions

Capabilities

skillsource-microsoftdocsskill-azure-databrickstopic-agenttopic-agent-skillstopic-agentic-skillstopic-agentskilltopic-ai-agentstopic-ai-codingtopic-azuretopic-azure-functionstopic-azure-kubernetes-servicetopic-azure-openaitopic-azure-sql-databasetopic-azure-storage

Install

Installnpx skills add MicrosoftDocs/Agent-Skills
Transportskills-sh
Protocolskill

Quality

0.70/ 1.00

deterministic score 0.70 from registry signals: · indexed on github topic:agent-skills · 549 github stars · SKILL.md body (61,499 chars)

Provenance

Indexed fromgithub
Enriched2026-05-18 18:53:52Z · deterministic:skill-github:v1 · v1
First seen2026-04-18
Last seen2026-05-18

Agent access