Daily Databricks

There's a really rich #R package for databricks from databricks-labs. You should give it ago. If you've previously used Sparklyr or SparkR this is a worth while package to pick up for your platform. I've jumped into a quick overview below dailydatabricks.tips/Tutoria…

Did you know about With Columns Renamed? You can pass in Key Vaule Pairs of {old_name: new_name} dailydatabricks.tips/tips/Sp…

Ever fancied using a managed identity for your #Azure #Databricks workspace that already exists and to start using it only requires a few lines of code? Find out how to use this from your Managed Resource Group dailydatabricks.tips/tips/Id…

1/2 Are verbose window functions bogging down your @databricks SQL & PySpark code? 😰 Named window specs to the rescue! 🚀

💡#Decembricks Day 2 Use #DeltaLake's custom metadata to: - Track data quality metrics - Maintain audit trails - Document pipeline versions - Monitor data transformations Full guide with code examples below 👇 @Databricks #DataEngineering #Databricks

4/6 Beginner-friendly? Absolutely! Just 3 lines of code to get started:

🎉 Great news! You can now add nicknames to your Databricks accounts! 🚀 Perfect for easily identifying your account in the UI, especially if managing multiple. Settings ➡️ Account settings tab ➡️ Enter nickname ➡️ Hit Save! You can update anytime. ✨ #Azure @databricks

Using Legacy Cluster init Scripts which are now deprecated? There's a Migration solution provided by @databricks kb.databricks.com/cluster-na…

The @MSPowerBI @databricks & @AzureDatabricks connector now supports Native SQL queries. You can read about this here 🔗databricks.com/blog/2023/01/…

Working with Streaming Data in @databricks? You can now define watermarks as of Runtime 12.X using the following docs.databricks.com/sql/lang… ❓Watermarks enable thresholds for processing updates in streaming. e.g. For use in Window Aggregation. Info here🔗learn.microsoft.com/en-us/az…

@Databricks recently introduced 🕑IngestionTimeClustering which helps take the pain out of common partitioning patterns. Good Delta Lake Design doesn't usually follow the conventional batch date partitioning Great article 🔗learn.microsoft.com/en-us/az…

🚨New Feature - You can create notification destinations for a variety of systems from @SlackHQ , Email, @MicrosoftTeams #databricks

Tips consolidated into a single image

🚨New feature in @Databricks Runtime 11.3+ You can incrementally clone Parquet and @ApacheIceberg tables to @DeltaLakeOSS It replicates the Delta Clone and Convert Tables to Delta Functionality Really helpful documentation on when and how to use 🔗learn.microsoft.com/en-us/az…

Want to achieve this programmatically via APIs? You can specify Cluster Init scripts on your @databricks workspace as part of your IaaC setups.

#Cluster Init Scripts Need to configure your @databricks cluster with some custom start up scripts? You can achieve this by putting them into the DBFS file store and referencing them within the cluster Init Script section teapaw.org/DailyDatabricks/status…

Create a Bash Script to be stored and executed from #databricks with dbutils Sample and example 👇

🚨New Feature: @Databricks Runtime 11.2 and above supports default values in tables for CSV, JSON, PARQUET & ORC. For those of you not on the #deltalake hype train that need this functionality see 👇 🔗docs.databricks.com/spark/la… #databricks #spark #dataengineering

Want to Block deletes and updates in a Delta table You can use the table property delta.appendOnly #databricks #delta #spark

🚨New Feature Bonus Round! When this was originally tweeted support for a datetime picker wasn't possible so you were limited to only dates! Now you can use the ipydatetime library to achieve this 👏 🔗ipydatetime.readthedocs.io/e… #Databricks @databricks #DataEngineering