Data Swindon

For this hybrid meeting, please register here so we can know who is attending in person

Agenda
18:00 Arrival, refreshments, and networking
18:30 Easy Data Pipelines with Databricks Delta Live Tables - Niall Langley MVP
19:20 Comfort Break
19:35 All About SQL Managed Instance - Richard Munn
20:25 Wrap-up
Please note: the event will not be catered, but feel free to bring your own snacks. Tea / Coffee / Water will be available. After the event, you're welcome to join us for an informal social at The Goddard Arms in Swindon Old Town.
Please let us know if you're attending in-person on registration as security passes need to be arranged the day before.
Presentations:

# 1. Easy Pipelines with Databricks Delta Live Tables

Delta Live Tables is a new framework available in Databricks that aims to accelerate building data pipelines by providing out of the box scheduling, dependency resolution, data validation and logging.
We'll cover the basics, and then get into the demo's to show how we can:
- Setup a notebook to hold our code and queries
- Ingest quickly and easily into bronze tables using Auto Loader
- Create views and tables on top of the ingested data using SQL and/or python to build our silver and gold layers
- Create a pipeline to run the notebook
- See how we can run the pipeline as either a batch job, or as a continuous job for low latency updates
- Use APPLY CHANGES INTO to upsert changed data into a live table
- Apply data validation rules to our live table definition queries, and get detailed logging info on how many records caused problems on each execution.
By the end of the session you should have a good view of whether this can help you build our your next data project faster, and make it more reliable.

# 2. All about SQL Managed Instances

Well, maybe not *all*, but there's definitely some key take-aways :-)
Azure SQL Managed Instances have been available for a few years now, and the features they provide just keep getting bigger and better.
In this session, we'll take a look at what they are, how they work, some of the benefits (and drawbacks) of using them, some obvious use-cases, and one that's *really* clever !