Skillquality 0.70

azure-databricks

Expert knowledge for Azure Databricks development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when using Unity Catalog, Lakehouse/Lakebase

Price
free
Protocol
skill
Verified
no

What it does

Azure Databricks Skill

This skill provides expert guidance for Azure Databricks. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g., L35-L120), use read_file with the specified lines. For categories with file links (e.g., [security.md](security.md)), use read_file on the linked reference file

IMPORTANT for Agent: If metadata.generated_at is more than 3 months old, suggest the user pull the latest version from the repository. If mcp_microsoftdocs tools are not available, suggest the user install it: Installation Guide

This skill requires network access to fetch documentation content:

  • Preferred: Use mcp_microsoftdocs:microsoft_docs_fetch with query string from=learn-agent-skill. Returns Markdown.
  • Fallback: Use fetch_webpage with query string from=learn-agent-skill&accept=text/markdown. Returns Markdown.

Category Index

CategoryLocationDescription
TroubleshootingL37-L138Diagnosing and fixing Databricks errors, job and compute failures, connector/ingestion issues, SQL error codes, and performance/debugging problems across Spark, AI, Lakeflow, and tooling.
Best PracticesL139-L313End-to-end Databricks best practices for performance, cost, governance, streaming, ML/LLM/RAG, BI, Lakeflow, Vector Search, and operational reliability across Azure Databricks workloads.
Decision MakingL314-L401Guides for choosing Azure Databricks architectures, compute, runtimes, ML/LLM options, and detailed migration paths (Unity Catalog, Delta, SQL, Connect, MLflow, serverless, Lakebase, and networking).
Architecture & Design PatternsL402-L444Architectural blueprints and patterns for Databricks: lakehouse, networking, storage, HA/DR, governance, performance, ML/MLOps, RAG/agents, Lakebase, streaming, and external data access.
Limits & Quotaslimits-quotas.mdLimits, quotas, and constraints for Databricks compute, AI/BI, connectors, Lakeflow, Lakebase, model serving, tokens, data types, and Unity Catalog resources, plus related workarounds.
Securitysecurity.mdIdentity, access control, encryption, networking, compliance, and secure integrations for Azure Databricks, Unity Catalog, Lakeflow, Lakebase, apps, and external data sources.
Configurationconfiguration.mdConfiguring and administering Azure Databricks: accounts, workspaces, security, networking, compute, storage, jobs, ML/serving, Lakehouse/Unity Catalog, Lakeflow, apps, and system-table–based monitoring.
Integrations & Coding Patternsintegrations.mdPatterns and APIs for integrating Databricks with external data systems, tools, and AI/ML frameworks, plus detailed PySpark/SQL function references and Lakehouse Federation/streaming examples.
Deploymentdeployment.mdDeploying and operating Databricks apps, agents, models, jobs, and infrastructure using CI/CD, IaC, bundles, serving, Terraform, Git, and region/release planning.

Troubleshooting

TopicURL
Monitor Genie space activity with audit log querieshttps://learn.microsoft.com/en-us/azure/databricks/ai-bi/admin/audit
Troubleshoot Azure Databricks compute startup issueshttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/
Resolve Databricks classic compute termination error codeshttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/cluster-error-codes
Debug Spark applications using Databricks Spark UIhttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/debugging-spark-ui
Troubleshoot Apache Kafka usage on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/connect/streaming/kafka/faq
Audit and monitor Delta Sharing activity with Databricks logshttps://learn.microsoft.com/en-us/azure/databricks/delta-sharing/audit-logs
Troubleshoot common Delta Sharing access errorshttps://learn.microsoft.com/en-us/azure/databricks/delta-sharing/troubleshooting
Troubleshoot common Databricks CLI errors and issueshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/troubleshooting
Use Databricks app details for monitoring and troubleshootinghttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/view-app-details
Troubleshoot Databricks Connect for Python issueshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/troubleshooting
Troubleshoot Databricks Connect for Scala problemshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/troubleshooting
Troubleshoot common Databricks Terraform provider errorshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/terraform/troubleshoot
Resolve common issues with Databricks VS Code extensionhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/faqs
Troubleshoot Databricks VS Code extension errorshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/troubleshooting
Resolve ARITHMETIC_OVERFLOW errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/arithmetic-overflow-error-class
Handle CAST_INVALID_INPUT errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/cast-invalid-input-error-class
Diagnose DC_GA4_RAW_DATA_ERROR in GA4 connectorhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-ga4-raw-data-error-error-class
Understand DC_SFDC_API_ERROR in Databricks connectorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sfdc-api-error-error-class
Diagnose DC_SQLSERVER_ERROR in SQL Server connectorhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sqlserver-error-error-class
Understand DELTA_ICEBERG_COMPAT_V1_VIOLATION errorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/delta-iceberg-compat-v1-violation-error-class
Handle DIVIDE_BY_ZERO errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/divide-by-zero-error-class
Handle Azure Databricks named error conditionshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/error-classes
Fix EWKB_PARSE_ERROR geometry parsing issueshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkb-parse-error-error-class
Fix EWKT_PARSE_ERROR geometry parsing issueshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkt-parse-error-error-class
Resolve GEOJSON_PARSE_ERROR in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/geojson-parse-error-error-class
Address GROUP_BY_AGGREGATE errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/group-by-aggregate-error-class
Handle H3_INVALID_CELL_ID errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-cell-id-error-class
Interpret and resolve H3_INVALID_GRID_DISTANCE_VALUE in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-grid-distance-value-error-class
Handle H3_INVALID_RESOLUTION_VALUE errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-resolution-value-error-class
Resolve H3_NOT_ENABLED errors and tier requirementshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-not-enabled-error-class
Fix INSUFFICIENT_TABLE_PROPERTY errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/insufficient-table-property-error-class
Troubleshoot INVALID_ARRAY_INDEX errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-error-class
Troubleshoot INVALID_ARRAY_INDEX_IN_ELEMENT_AT in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-in-element-at-error-class
Resolve MISSING_AGGREGATION errors in Databricks querieshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/missing-aggregation-error-class
Diagnose ROW_COLUMN_ACCESS errors for filters and maskshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/row-column-access-error-class
Interpret Azure Databricks SQLSTATE error codeshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/sqlstates
Fix TABLE_OR_VIEW_NOT_FOUND errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/table-or-view-not-found-error-class
Resolve UNRESOLVED_ROUTINE function resolution errorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/unresolved-routine-error-class
Understand UNSUPPORTED_TABLE_OPERATION errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-table-operation-error-class
Understand UNSUPPORTED_VIEW_OPERATION errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-view-operation-error-class
Troubleshoot WKB_PARSE_ERROR for geometry parsinghttps://learn.microsoft.com/en-us/azure/databricks/error-messages/wkb-parse-error-error-class
Troubleshoot WKT_PARSE_ERROR for geometry parsinghttps://learn.microsoft.com/en-us/azure/databricks/error-messages/wkt-parse-error-error-class
Troubleshoot Mosaic AI Agent Evaluation issueshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/troubleshooting
Troubleshoot and debug Databricks AI agent deploymentshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/debug-agent
Troubleshoot common Azure Databricks Genie issueshttps://learn.microsoft.com/en-us/azure/databricks/genie/troubleshooting
Resolve common Databricks Auto Loader questions and issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/faq
Diagnose and fix Databricks Confluence ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/confluence-troubleshoot
Troubleshoot Dynamics 365 data ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/d365-troubleshoot
Troubleshoot Google Ads connector ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-ads-troubleshoot
Troubleshoot Google Analytics raw data ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-analytics-troubleshoot
Troubleshoot HubSpot connector ingestion problemshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/hubspot-troubleshoot
Troubleshoot Jira Lakeflow ingestion errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/jira-troubleshoot
Troubleshoot Meta Ads Lakeflow ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/meta-ads-troubleshoot
Diagnose and fix MySQL Lakeflow Connect ingestion errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/mysql-troubleshoot
Resolve common PostgreSQL Lakeflow Connect connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-faq
Troubleshoot PostgreSQL ingestion with Lakeflow Connecthttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-troubleshoot
Troubleshoot Lakeflow Connect query-based connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/query-based-troubleshoot
Troubleshoot Salesforce Lakeflow ingestion problemshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-troubleshoot
Diagnose and fix Databricks ServiceNow connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/servicenow-troubleshoot
Diagnose and fix Lakeflow SharePoint connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sharepoint-troubleshoot
Answer common SQL Server Lakeflow Connect connector questionshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-faq
Troubleshoot SQL Server ingestion with Lakeflow Connecthttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-troubleshoot
Troubleshoot TikTok Ads connector in Lakeflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/tiktok-ads-troubleshoot
Troubleshoot Workday HCM connector in Lakeflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-hcm-troubleshoot
Diagnose and fix Databricks Workday connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-reports-troubleshoot
Troubleshoot Databricks Zendesk Support connector errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/zendesk-support-troubleshoot
Handle Zerobus Ingest errors and retrieshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-errors
Use logging to troubleshoot Databricks init scriptshttps://learn.microsoft.com/en-us/azure/databricks/init-scripts/logs
Test and validate legacy Databricks JDBC driver connectionshttps://learn.microsoft.com/en-us/azure/databricks/integrations/jdbc/testing
Test and validate Databricks ODBC driver connectionshttps://learn.microsoft.com/en-us/azure/databricks/integrations/odbc/testing
Configure and troubleshoot Lakeflow Jobs with many taskshttps://learn.microsoft.com/en-us/azure/databricks/jobs/large-jobs
Troubleshoot and repair Azure Databricks job failureshttps://learn.microsoft.com/en-us/azure/databricks/jobs/repair-job-failures
Monitor and troubleshoot Lakeflow Spark pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/observability
Use pipeline query history for debugging and tuninghttps://learn.microsoft.com/en-us/azure/databricks/ldp/query-history
Recover Databricks pipelines from checkpoint failureshttps://learn.microsoft.com/en-us/azure/databricks/ldp/recover-streaming
User guides, migration, and troubleshooting for AI Runtimehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ai-runtime/guides
Troubleshoot Databricks Feature Store issues and limitshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/troubleshooting-and-limitations
Debug common issues in Databricks Model Serving endpointshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-debug
Diagnose Databricks model serving issues with Genie Codehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-genie-code
Debug Python code in Databricks notebookshttps://learn.microsoft.com/en-us/azure/databricks/notebooks/debugger
Troubleshoot failing Spark jobs and executors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/failing-spark-jobs
Use Databricks Spark jobs timeline for debugginghttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/jobs-timeline
Diagnose long-running Spark stages in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage
Investigate high I/O Spark stages in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage-io
Debug slow low-I/O Spark stages in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/slow-spark-stage-low-io
Identify expensive reads in Spark DAG on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-dag-expensive-read
Diagnose gaps between Spark jobs in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-job-gaps
Diagnose and fix Spark memory issues on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-memory-issues
Troubleshoot Azure Databricks Partner Connect issueshttps://learn.microsoft.com/en-us/azure/databricks/partner-connect/troubleshoot
Troubleshoot Databricks Git folder errorshttps://learn.microsoft.com/en-us/azure/databricks/repos/errors-troubleshooting
Fetch cursor rows and handle SQLSTATE in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/fetch-stmt
Use GET DIAGNOSTICS for SQL error handling in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/get-diagnostics-stmt
Open cursors and handle errors with OPEN in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/open-stmt
Validate UTF-8 strings and handle INVALID_UTF8_STRINGhttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/validate_utf8
Understand Databricks SQL query performance insightshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/performance-insights
Use Databricks query history to debug performancehttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-history
Interpret Databricks query profiles for performance tuninghttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-profile
Identify and clean up unused Vector Search endpointshttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-unused-endpoints

Best Practices

TopicURL
Use Databricks default compute policy families effectivelyhttps://learn.microsoft.com/en-us/azure/databricks/admin/clusters/policy-families
Apply identity best practices and migrate to federationhttps://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/best-practices
Apply best practices for Azure Databricks serverless workspaceshttps://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces-best-practices
Migrate Databricks library installs from init scriptshttps://learn.microsoft.com/en-us/azure/databricks/archive/compute/libraries-init-scripts
Apply best practices for Databricks compute policieshttps://learn.microsoft.com/en-us/azure/databricks/archive/compute/policies-best-practices
Use DBIO for transactional writes to cloud storage in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/archive/legacy/dbio-commit
Optimize skewed joins in Databricks using skew hintshttps://learn.microsoft.com/en-us/azure/databricks/archive/legacy/skew-join
Migrate from Databricks Deep Learning Pipelineshttps://learn.microsoft.com/en-us/azure/databricks/archive/spark-3.x-migration/deep-learning-pipelines
Model business data with Unity Catalog metric viewshttps://learn.microsoft.com/en-us/azure/databricks/business-semantics/metric-views/basic-modeling
Apply Azure Databricks platform administration best practiceshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/administration
Optimize BI performance with Databricks SQL warehouseshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving
Prepare and model data for high-performance BI on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-data-prep
Configure Databricks SQL warehouses for optimal BI servinghttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-sql-serving
Apply Azure Databricks compute creation best practiceshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/compute
Implement Azure Databricks production job scheduling best practiceshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/jobs
Best practices for Power BI dashboards on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/power-bi
Apply Databricks compute configuration best practiceshttps://learn.microsoft.com/en-us/azure/databricks/compute/cluster-config-best-practices
Use flexible node types for reliable Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/flexible-node-types
Apply best practices for Databricks poolshttps://learn.microsoft.com/en-us/azure/databricks/compute/pool-best-practices
Apply serverless compute best practices in Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/compute/serverless/best-practices
Tune Databricks SQL warehouses for BI workloadshttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/bi-workload-settings
Use system table queries to monitor Databricks SQL warehouseshttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/monitor/queries
Control large interactive queries with Query Watchdoghttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/query-watchdog
Apply observability best practices for Databricks jobs and pipelineshttps://learn.microsoft.com/en-us/azure/databricks/data-engineering/observability-best-practices
Write efficient UDFs for Unity Catalog ABAC policieshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/udf-best-practices
Apply Unity Catalog data governance best practiceshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/best-practices
Work with legacy Hive metastore database objectshttps://learn.microsoft.com/en-us/azure/databricks/database-objects/hive-metastore
Follow DBFS root storage recommendations in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/dbfs/dbfs-root
Migrate from DBFS mounts to Unity Catalog external locationshttps://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts
Apply DBFS and Unity Catalog usage best practiceshttps://learn.microsoft.com/en-us/azure/databricks/dbfs/unity-catalog
Optimize Delta Sharing to reduce cloud egress costshttps://learn.microsoft.com/en-us/azure/databricks/delta-sharing/manage-egress
Apply Delta Lake best practices on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/best-practices
Optimize Databricks tables with liquid clusteringhttps://learn.microsoft.com/en-us/azure/databricks/delta/clustering
Tune Azure Databricks data skipping with stats and Z-orderhttps://learn.microsoft.com/en-us/azure/databricks/delta/data-skipping
Use deletion vectors to accelerate Delta table modifications on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/deletion-vectors
Optimize Delta table file layout on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/optimize
Handle Delta Lake limitations on Amazon S3https://learn.microsoft.com/en-us/azure/databricks/delta/s3-limitations
Apply selective overwrite patterns in Delta Lakehttps://learn.microsoft.com/en-us/azure/databricks/delta/selective-overwrite
Control Delta table data file size on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/tune-file-size
Vacuum Delta tables safely and efficiently on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/vacuum
Optimize VARIANT data performance with shredding on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/variant-shredding
Apply MLOps Stack best practices with bundleshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/mlops-stacks
Apply Databricks-recommended CI/CD workflows and patternshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/best-practices
View Databricks cluster policy families via CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/policy-families-commands
Apply security and performance best practices for Databricks appshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/best-practices
Test Databricks Connect for Python code with pytesthttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/testing
Handle async queries and interruptions in Databricks Connecthttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/queries
Choose between Databricks volumes and workspace fileshttps://learn.microsoft.com/en-us/azure/databricks/files/files-recommendations
Customize MLflow 2 AI judges for your agentshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/advanced-agent-eval
Apply best practices for MLflow 2 evaluation setshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/evaluation-set
Measure RAG performance with Databricks metricshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/evaluate-assess-performance
Create evaluation sets for Databricks RAG appshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/evaluate-define-quality
Evaluate and monitor RAG apps on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/fundamentals-evaluation-monitoring-rag
Optimize Databricks RAG application qualityhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-overview
Improve Databricks RAG chain qualityhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain
Configure Genie Code custom instructions effectivelyhttps://learn.microsoft.com/en-us/azure/databricks/genie-code/instructions
Apply practical tips to improve Genie Code responseshttps://learn.microsoft.com/en-us/azure/databricks/genie-code/tips
Apply best practices for curating Genie spaceshttps://learn.microsoft.com/en-us/azure/databricks/genie/best-practices
Migrate existing Auto Loader streams to file eventshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/migrating-to-file-events
Apply common Auto Loader data ingestion patternshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/patterns
Configure Databricks Auto Loader for productionhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/production
Configure Auto Loader with Unity Catalog for secure ingestionhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/unity-catalog
Apply common COPY INTO data loading patternshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/copy-into/examples
Use the _metadata file column in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/file-metadata-column
Apply common patterns for Lakeflow managed ingestion pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/common-patterns
Fully refresh Lakeflow Connect managed ingestion target tableshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/full-refresh
Query system.billing.usage to monitor Lakeflow ingestion costshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/monitor-costs
Perform ongoing maintenance for Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/pipeline-maintenance
Maintain and operate PostgreSQL ingestion pipelines in Lakeflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-maintenance
Enable incremental ingestion for Salesforce formula fieldshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-formula-fields
Use Databricks init scripts for cluster customizationhttps://learn.microsoft.com/en-us/azure/databricks/init-scripts/
Reference external files safely in Databricks init scriptshttps://learn.microsoft.com/en-us/azure/databricks/init-scripts/referencing-files
Configure compute for Lakeflow Jobs with recommended patternshttps://learn.microsoft.com/en-us/azure/databricks/jobs/compute
Build metadata-driven For each jobs with control tableshttps://learn.microsoft.com/en-us/azure/databricks/jobs/how-to/foreach-sql-lookup-tutorial
Apply best practices for configuring classic Lakeflow Jobshttps://learn.microsoft.com/en-us/azure/databricks/jobs/run-classic-jobs
Apply cost optimization best practices on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/cost-optimization/best-practices
Implement best practices for Databricks data and AI governancehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/best-practices
Design observability and monitoring strategy for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/observability
Apply interoperability and usability best practices on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/interoperability-and-usability/best-practices
Implement operational excellence best practices on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/operational-excellence/best-practices
Apply performance best practices for Azure Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/performance-efficiency/best-practices
Apply reliability best practices on Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reliability/best-practices
Optimize Lakeflow pipelines with enhanced autoscalinghttps://learn.microsoft.com/en-us/azure/databricks/ldp/auto-scaling
Apply best practices for Lakeflow Spark Declarative Pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/best-practices
Use advanced AUTO CDC features and monitor processing metricshttps://learn.microsoft.com/en-us/azure/databricks/ldp/cdc-advanced
Apply development best practices to Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/develop
Manage Python dependencies safely in Databricks pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/developer/external-dependencies
Implement advanced expectation patterns at scalehttps://learn.microsoft.com/en-us/azure/databricks/ldp/expectation-patterns
Reduce Lakeflow pipeline initialization latencyhttps://learn.microsoft.com/en-us/azure/databricks/ldp/fix-high-init
Backfill historical data with Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/flows-backfill
Run full refresh operations for Databricks streaming tables safelyhttps://learn.microsoft.com/en-us/azure/databricks/ldp/full-refresh-st
Optimize stateful stream processing with watermarkshttps://learn.microsoft.com/en-us/azure/databricks/ldp/stateful-processing
Design CDC and snapshot patterns in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/ldp/what-is-change-data-capture
Restart the Python process to refresh Databricks librarieshttps://learn.microsoft.com/en-us/azure/databricks/libraries/restart-python-process
Apply data loading best practices on Databricks AI Runtimehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ai-runtime/dataloading
Track experiments and monitor GPU health on AI Runtimehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ai-runtime/tracking-observability
Apply Hyperopt best practices and troubleshooting on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/automl-hyperparam-tuning/hyperopt-best-practices
Implement point-in-time correct feature joins for time serieshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/time-series
Benchmark Databricks LLM endpoints for latency and throughputhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/foundation-model-apis/prov-throughput-run-benchmark
Load and prepare data for ML on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/load-data/
Implement LLMOps workflows on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/llmops
Configure Locust-based load tests for Databricks servinghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/configure-load-test
Validate models before Databricks Model Serving deploymenthttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-pre-deployment-validation
Optimize Databricks Model Serving endpoints for productionhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/production-optimization
Plan and execute load testing for Databricks endpointshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/what-is-load-test
Tune and scale Ray clusters on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ray/scale-ray
Implement distributed image inference on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/reference-solutions/images-etl-inference
Follow deep learning best practices on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/dl-best-practices
Fine-tune Hugging Face models on a single GPU in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/huggingface/fine-tune-model
Prepare datasets for Hugging Face fine-tuning on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/huggingface/load-data
Adapt Apache Spark workloads for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/migration/spark
Align MLflow LLM judges with human evaluatorshttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/align-judges
Optimize prompts using MLflow GEPA optimizerhttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/prompt-version-mgmt/prompt-registry/automatically-optimize-prompts
Evaluate and compare MLflow prompt versions effectivelyhttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/prompt-version-mgmt/prompt-registry/evaluate-prompts
Use manual MLflow tracing for production GenAI appshttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/tracing/app-instrumentation/manual-tracing/
Analyze GenAI traces for errors and performancehttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/tracing/observe-with-traces/analyze-traces
Apply software engineering practices to Databricks notebookshttps://learn.microsoft.com/en-us/azure/databricks/notebooks/best-practices
Apply Genie Code effectively in Databricks notebookshttps://learn.microsoft.com/en-us/azure/databricks/notebooks/code-assistant
Run Databricks notebooks safely and efficientlyhttps://learn.microsoft.com/en-us/azure/databricks/notebooks/run-notebook
Test and schedule Databricks notebook codehttps://learn.microsoft.com/en-us/azure/databricks/notebooks/test-notebooks
Configure and optimize Lakebase Postgres computeshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/manage-computes
Create and manage Lakebase read replicashttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/manage-read-replicas
Monitor Lakebase queries using pg_stat_statementshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/pg-stat-statements
Apply performance optimization recommendations on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/
Use adaptive query execution on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/aqe
Leverage cost-based optimizer in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/cbo
Improve read performance with Databricks disk cachehttps://learn.microsoft.com/en-us/azure/databricks/optimizations/disk-cache
Improve Delta query performance with dynamic file pruning on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/dynamic-file-pruning
Accelerate data access with predictive I/Ohttps://learn.microsoft.com/en-us/azure/databricks/optimizations/predictive-io
Use predictive optimization for Unity Catalog tableshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/predictive-optimization
Tune Azure Databricks range join performancehttps://learn.microsoft.com/en-us/azure/databricks/optimizations/range-join
Diagnose Databricks Spark cost and performance in UIhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/
Debug skew and spill in Databricks Spark stageshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage-page
Handle Databricks spot instance losses effectivelyhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/losing-spot-instances
Resolve long Spark stages with a single taskhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/one-spark-task
Optimize many small Spark jobs on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/small-spark-jobs
Mitigate overloaded Spark driver on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-driver-overloaded
Detect unnecessary data rewriting in Databricks Spark writeshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-rewriting-data
Best practices for setting up Databricks Partner Connecthttps://learn.microsoft.com/en-us/azure/databricks/partner-connect/best-practice
Configure networking for Databricks Lakehouse Federation data sourceshttps://learn.microsoft.com/en-us/azure/databricks/query-federation/networking
Optimize performance of Databricks Lakehouse Federation querieshttps://learn.microsoft.com/en-us/azure/databricks/query-federation/performance-recommendations
Encrypt inter-node traffic in Databricks clustershttps://learn.microsoft.com/en-us/azure/databricks/security/keys/encrypt-otw
Transform complex and nested data types in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/complex-types
Use higher-order functions on arrays in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/higher-order-functions
Query semi-structured data using VARIANT in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/variant
Differences between VARIANT and JSON strings in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/variant-json-diff
Work with OBJECT type and VARIANT schemas in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/object-type
Use TIMESTAMP_NTZ type and Delta support in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/timestamp-ntz-type
Use VARIANT type and Iceberg compatibility in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/variant-type
Collect table statistics with ANALYZE TABLE for optimizationhttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-aux-analyze-compute-statistics
Use Databricks SQL query caching for performancehttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-caching
Use Databricks SQL query filters effectivelyhttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-filters
Optimize queries using primary key constraints in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-optimization-constraints
Use Delta tables for streaming reads and writeshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/delta-lake
Production best practices for Databricks Structured Streaminghttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/production
Optimize and monitor Databricks real-time streaming performancehttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/real-time/performance
Optimize stateless Structured Streaming queries on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/stateless-streaming
Monitor Azure Databricks Structured Streaming querieshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/stream-monitoring
Combine Unity Catalog with Structured Streaming workloadshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/unity-catalog
Apply watermarks for efficient stateful streaminghttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/watermarks
Use Automatic Feature Enablement for Unity Catalog tableshttps://learn.microsoft.com/en-us/azure/databricks/tables/automatic-feature-enablement
Analyze and optimize Delta table storage size on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/tables/size
Design data models optimized for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/transform/data-modeling
Optimize join performance for Azure Databricks workloadshttps://learn.microsoft.com/en-us/azure/databricks/transform/optimize-joins
Clean and validate data with Databricks batch and streaminghttps://learn.microsoft.com/en-us/azure/databricks/transform/validate
Evaluate and compare Mosaic AI Vector Search retrieval qualityhttps://learn.microsoft.com/en-us/azure/databricks/vector-search/retrieval-quality-eval
Optimize Mosaic AI Vector Search performancehttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-best-practices
Optimize and manage Mosaic AI Vector Search costshttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-cost-management
Design and run load tests for Vector Search endpointshttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-endpoint-load-test
Improve Mosaic AI Vector Search retrieval qualityhttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-retrieval-quality
Download internet data into Azure Databricks volumeshttps://learn.microsoft.com/en-us/azure/databricks/volumes/download-internet-files

Decision Making

TopicURL
Manage and change Azure Databricks subscription planshttps://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/account
Plan migration from Databricks Standard to Premium tierhttps://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/standard-tier
Decide when and how to use serverless workspaceshttps://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces
Decide and migrate from dbx to Databricks bundleshttps://learn.microsoft.com/en-us/azure/databricks/archive/dev-tools/dbx/dbx-migrate
Migrate optimized LLM endpoints to provisioned throughputhttps://learn.microsoft.com/en-us/azure/databricks/archive/machine-learning/migrate-provisioned-throughput
Decide when to use Databricks Light runtimehttps://learn.microsoft.com/en-us/azure/databricks/archive/runtime/light
Plan migration of Databricks workloads to Spark 3.xhttps://learn.microsoft.com/en-us/azure/databricks/archive/spark-3.x-migration/
Select and manage the default Unity Catalog cataloghttps://learn.microsoft.com/en-us/azure/databricks/catalogs/default
Choose appropriate Azure Databricks compute typeshttps://learn.microsoft.com/en-us/azure/databricks/compute/choose-compute
Decide when and how to use GPU Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/gpu
Decide when and how to use Azure Databricks poolshttps://learn.microsoft.com/en-us/azure/databricks/compute/pool-index
Plan migration from classic to serverless Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/serverless/migration
Plan Databricks SQL warehouse sizing and queuinghttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/warehouse-behavior
Choose between Databricks SQL warehouse typeshttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/warehouse-types
Choose managed vs external assets in Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/managed-versus-external
Plan and execute upgrade of Databricks workspaces to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/upgrade/
Prepare and migrate to Unity Catalog–only Databricks workspaceshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/upgrade/uc-only-migration
Choose Delta Lake protocol versions and feature setshttps://learn.microsoft.com/en-us/azure/databricks/delta/feature-compatibility
Choose local development tools for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/
Migrate from legacy to new Databricks CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/migrate
Manage Databricks account budget policies via CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/account-budget-policy-commands
Configure Databricks account budgets using CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/account-budgets-commands
Manage Databricks account usage dashboards via CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/account-usage-dashboards-commands
Select and configure compute size for Databricks Appshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/compute-size
Plan migration from legacy Databricks Connect runtimeshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect-legacy
Migrate from older to new Databricks Connect for Pythonhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/migrate
Migrate from legacy to new Scala Databricks Connecthttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/migrate
Choose and use Databricks SDKs for automationhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/sdks
Decide between CDKTF and Databricks Terraform providerhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/terraform/cdktf
Choose Unity Catalog integration method for external engineshttps://learn.microsoft.com/en-us/azure/databricks/external-access/integrations
Interpret MLflow 2 Agent Evaluation quality, cost, latencyhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/llm-judge-metrics
Migrate Databricks Community Edition to Free Editionhttps://learn.microsoft.com/en-us/azure/databricks/getting-started/ce-migration
Choose between Databricks Free Edition and free trialhttps://learn.microsoft.com/en-us/azure/databricks/getting-started/free-trial-vs-free-edition
Choose incremental ingestion options from cloud object storagehttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/
Select Auto Loader file detection mode for your workloadhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/file-detection-modes
Plan migration of existing data to Delta Lake on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/data-migration/
Plan MySQL ingestion workflow and setup in Lakeflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/mysql
Choose and start with Databricks ODBC and JDBC drivershttps://learn.microsoft.com/en-us/azure/databricks/integrations/jdbc-odbc-bi
Migrate from Simba Spark ODBC to Databricks ODBChttps://learn.microsoft.com/en-us/azure/databricks/integrations/odbc/migration
Plan and manage production workloads with Lakeflow Jobshttps://learn.microsoft.com/en-us/azure/databricks/jobs/
Decide when to run Lakeflow Jobs on serverless computehttps://learn.microsoft.com/en-us/azure/databricks/jobs/run-serverless-jobs
Migrate from Spark Submit tasks in Databricks jobshttps://learn.microsoft.com/en-us/azure/databricks/jobs/spark-submit
Plan production Azure Databricks lakehouse deploymentshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/
Design compute and workspace configuration for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/compute
Choose a programming language for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/languages/overview
Migrate legacy and third-party online tables to Lakebasehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/migrate-from-online-tables
Upgrade workspace feature tables to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/uc/upgrade-feature-table-to-uc
Choose Databricks-hosted foundation models and endpointshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/foundation-model-apis/supported-models
Migrate MLflow model versions to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/migrate-models
Decide and migrate to Unity Catalog model managementhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/migrate-to-uc
Upgrade Databricks ML workflows to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/upgrade-workflows
Choose Databricks options for batch model inferencehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-inference/
Select supported foundation models on Mosaic AIhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/foundation-model-overview
Migrate from legacy MLflow to Mosaic AI Model Servinghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/migrate-model-serving
Decide when to use Spark vs. Ray on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ray/spark-ray-overview
Plan migration of data applications to Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/migration/
Assess options for migrating ETL pipelines to Databrickshttps://learn.microsoft.com/en-us/azure/databricks/migration/etl
Choose a migration path from Parquet to Delta Lakehttps://learn.microsoft.com/en-us/azure/databricks/migration/parquet-to-delta-lake
Migrate enterprise data warehouses to the Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/migration/warehouse-to-lakehouse
Decide and migrate from Agent Evaluation to MLflow 3https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/agent-eval-migration
Quick reference for migrating to MLflow 3https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/agent-eval-migration-reference
Choose between open source and Databricks MLflowhttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/overview/oss-managed-diff
Choose compute resources for Databricks notebookshttps://learn.microsoft.com/en-us/azure/databricks/notebooks/notebook-compute
Right-size Lakebase instance capacity and scalinghttps://learn.microsoft.com/en-us/azure/databricks/oltp/instances/create/capacity
Choose backup and restore methods for Lakebasehttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/backup-methods
Decide between Lakebase Provisioned and Autoscaling projectshttps://learn.microsoft.com/en-us/azure/databricks/oltp/upgrade-to-autoscaling
Choose and configure incremental refresh for Databricks materialized viewshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/incremental-refresh
Choose pandas options on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/pandas/
Plan and use Hive metastore federation with Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/query-federation/hms-federation-concepts
Migrate Databricks HTTP routing to serverless computehttps://learn.microsoft.com/en-us/azure/databricks/query-federation/http-migration
Migrate legacy Databricks query federation to Lakehouse Federationhttps://learn.microsoft.com/en-us/azure/databricks/query-federation/migrate
Plan and execute migration to Databricks Runtime 11.xhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/11.x-migration
Migrate workloads to Databricks Runtime 12.x safelyhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/12.x-migration
Migrate workloads to Databricks Runtime 13.x safelyhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/13.x-migration
Migrate workloads to Databricks Runtime 14.x safelyhttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/14.x-migration
Plan around Databricks Runtime support lifecycleshttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/databricks-runtime-ver
Understand serverless DBU billing by Azure Databricks SKUhttps://learn.microsoft.com/en-us/azure/databricks/resources/pricing
Evaluate Databricks serverless networking data transfer costshttps://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/cost-management
Decide between Spark Connect and Spark Classichttps://learn.microsoft.com/en-us/azure/databricks/spark/connect-vs-classic
Decide between SparkR and sparklyr on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sparkr/sparkr-vs-sparklyr
Migrate to the latest Databricks SQL REST APIhttps://learn.microsoft.com/en-us/azure/databricks/sql/dbsql-api-latest
Use SYNC to upgrade Hive tables to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-aux-sync
Choose Structured Streaming output modes on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/output-mode
Choose and implement Databricks transaction modeshttps://learn.microsoft.com/en-us/azure/databricks/transactions/transaction-modes

Architecture & Design Patterns

TopicURL
Plan disaster recovery architecture for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/admin/disaster-recovery
Design and use materialization for Databricks metric viewshttps://learn.microsoft.com/en-us/azure/databricks/business-semantics/metric-views/materialization
Implement fan-in and fan-out patterns in Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/data-engineering/fan-in-fan-out
Choose patterns for external access to Databricks datahttps://learn.microsoft.com/en-us/azure/databricks/external-access/
Build an IDP pipeline with Databricks AI Functionshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-bricks/idp-pipeline-tutorial
Build multi-agent orchestrator apps on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/multi-agent-apps
Create Genie-based multi-agent systems on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/multi-agent-genie
Build non-conversational Databricks AI agents with MLflowhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/non-conversational-agents
Implement AI agent memory on Databricks Model Servinghttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/stateful-agents-model-serving
Apply agent system design patterns on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/guide/agent-system-design-patterns
Design measurement infrastructure for RAG quality on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/evaluate-enable-measurement
Design and tune Databricks RAG inference chainshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/fundamentals-inference-chain-rag
Design cost optimization architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/cost-optimization/
Apply data and AI governance architecture on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/
Design Delta Lake and medallion data architecture on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/delta-lake
Design high availability and disaster recovery for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/ha-dr
Design Azure Databricks network and connectivity architecturehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/network
Design storage architecture for Azure Databricks and Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/storage
Design Azure Databricks workspace architecture strategyhttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/deployment-guide/workspace-strategy
Design interoperability and usability architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/interoperability-and-usability/
Design operational excellence architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/operational-excellence/
Design performance efficiency architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/performance-efficiency/
Apply Azure Databricks lakehouse reference architectureshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reference
Design reliability architecture for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reliability/
Apply the data lakehouse pattern on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse/
Design online feature workflows with Databricks and third-party storeshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/online-workflows
Choose Databricks ML model deployment patternshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/deployment-patterns
Implement MLOps workflows on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/mlops-workflow
Choose and train deep-learning recommenders in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-recommender-models
Use Lakebase branches for database development workflowshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/branches
Design for high availability with Lakebase computeshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/high-availability
Scale reads with Lakebase read replicashttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/read-replicas
Serve lakehouse data via Lakebase synced tableshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/sync-tables
Connect Databricks Serverless Private Git to on-prem Githttps://learn.microsoft.com/en-us/azure/databricks/repos/connect-on-prem-git-server
Set up Databricks Serverless Private Git with Private Linkhttps://learn.microsoft.com/en-us/azure/databricks/repos/serverless-private-git
Choose patterns for modeling semi-structured data on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/
Use asynchronous state checkpointing in Structured Streaminghttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/async-checkpointing
Apply asynchronous progress tracking in Structured Streaminghttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/async-progress-checking
Decide when to partition Delta tables on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/tables/partitions

Capabilities

skillsource-microsoftdocsskill-azure-databrickstopic-agenttopic-agent-skillstopic-agentic-skillstopic-agentskilltopic-ai-agentstopic-ai-codingtopic-azuretopic-azure-functionstopic-azure-kubernetes-servicetopic-azure-openaitopic-azure-sql-databasetopic-azure-storage

Install

Installnpx skills add MicrosoftDocs/Agent-Skills
Transportskills-sh
Protocolskill

Quality

0.70/ 1.00

deterministic score 0.70 from registry signals: · indexed on github topic:agent-skills · 497 github stars · SKILL.md body (62,386 chars)

Provenance

Indexed fromgithub
Enriched2026-04-22 06:53:31Z · deterministic:skill-github:v1 · v1
First seen2026-04-18
Last seen2026-04-22

Agent access