Archives

How to Import Existing SSIS Packages into Biml With BimlExpress

I’m asked the same question almost every time I present on Biml. “How do I import my existing SSIS packages?” Well in this post, I’m going to show you how to do it with BimlExpress 2019 for free.

This post is part of a series, Short Cuts to Biml Mastery which shows you how to learn BimlScript through a series of short cuts and remain productive while doing so.

What is BimlExpress?

BimlExpress is a free Visual Studio add-in created by the good folks at Varigence. Its a full featured Biml editor which allows you to dynamically create SSIS packages. It was first released back in 2017, and the latest version is 2019 (of course). The current version supports Visual Studio 2010 through 2019 as well as SQL Server 2005 through 2019.

Prior to it’s release, Biml was written with Bids Helper, now known as BI Developer Extensions. While BI Developer Extensions has many nice features, you should no longer use it to work with Biml as it is no longer being updated/supported.

How do I use it?

Well, first download the BimlExpress VSIX install file, run it, and follow the prompts. Next, please download the Northwind sample database and restore it to your development machine. Finally, download this VS Sample Project. As I mentioned previously, this simple package forms the basis for the next several blog posts. Open the package, you can see that it exports from a SQL Server table to a csv flat file. Edit the connection details if necessary, and create the location for the export file. The default location is C:\FileExport\.

SSIS Data Flow
Flat File General Tab

Flat File Advanced Tab

Most of the functionality of BimlExpress can be found in the custom toolbar. For our demo, make sure the Export Orders.dtsx package is selected, and then choose “Convert SSIS Packages to Biml.”

You can also right-click either the package or the entire project in the Solution Explorer to import as well.

Import Packages

This is the Import Packages window. There are four options for SSIS Asset Location:

  • DTProj from File System
  • ISPAC from File System
  • DTSX from File System
  • Project from Project Catalog

Select “DTSX from File System” since we are importing a single package. There are all sorts of Import Options which I don’t want to take the time to go into. Most often, I leave all of them unchecked anyway.

Imported Assets

Select Import and examine the imported assets. Everything looks correct, so click “Add to Project”.

Imported Biml

As you can see in the above screen shot, BimlExpress has made great strides in usability. Not only can you import packages, but you can edit BimlScript and see the results immediately in the Preview Pane. Notice that all of the Biml is imported into one file. We’ll examine and simplify the imported Biml in my next post.

To test your Biml file, edit the Package name to something different like “Export Orders Biml”. Save the change, right click on BimlScript.biml, and select “Generate SSIS Packages”. The package should generate and execute without error if you have the correct connection strings and such. Feel free to run the package and examine the resulting csv file. After the successful test, generate the package a second time and you’ll receive the following confirmation request to overwrite an existing package:

Confirm Overwrite

Next Steps

In the next article, we’ll examine our imported Biml code, clean it up, and start down the path of dynamic package generation.

Stay Tuned!

Short Cuts to Biml Mastery

I’ve been giving Biml related presentations for many years and shared what I believe are the most efficient ways to learn it. It’s possible to get enough Biml knowledge to become productive in a number of weeks, assuming one has time to focus on it. However context switching, commonly referred to as multi-tasking, kills the focus and concentration required to learn complex tasks. The unfortunate reality is that most employers can’t (or won’t) allow this kind of schedule flexibility.

So, how can you learn to apply Biml in a way that works within your schedule? Well, this series of articles will cover that very topic. I’ll show you some short cuts to generating Biml that will give you immediate productivity, so you can leverage “free” time to learn more.

In this series, we’ve been tasked by our boss to create several flat file extracts from a single database. The file specification is comma-separated values (CSV) format, double quote text qualified.

By the end of this series, we’ll have a completed, robust framework which will loop through a specified set of queries (or tables, views, etc.) and dynamically create an export for each. Additionally, it’ll support any
delimited flat file format.

Be sure to check back to this page for a list of articles as they are published.

Remembering Warren Thornthwaite

Warren Thornthwaite

I met Warren at my first PASS Summit (2010) after his “Dimensional Modeling Fundamentals” presentation. I think I embarrassed him a little when I gushed about how much I loved his books.

What? You didn’t know Warren? Well, let me tell you a bit about him. He was exceptionally kind, compassionate, brilliant, and generous with his time. He was a prolific presenter, consultant, and author. I’ve read all of the Kimball Group books in the past, but my favorite is The Microsoft Data Warehouse Toolkit. In fact, I was searching through that book yesterday, which is what prompted me to write this. Dimensional modeling is a timeless skill, and his books will forever impart this skill to those who seek to master it.

Warren was a master teacher and presenter. All of the Kimball Group presenters are/were excellent, but Warren was my favorite. He was brilliant, but self-effacing. I once saw him admit that a piece of code in his presentation wasn’t actually his, when an audience member asked a question about it. He was one of those rare people who was not afraid to say, “I don’t know.”

In 2013, I found myself involved in a very large and complicated data warehouse project. I had been thrust into the position of Architect and was in charge of a large technical team. This project had already been attempted and failed by several different teams. Anyway, I called in several experts to advise and work on the project and assure the business that the project was progressing efficiently. Since money was no object, I called in Warren to review and advise on the data model. Basically, he was there to check my homework. He spent several days on-site with us, and we all learned a great deal from him. I found him to be a masterful consultant as well. He didn’t look for “problems” just to justify his time. He wasn’t pushy or arrogant, even if someone asked him really basic questions. He didn’t lecture but rather relied more on the Socratic method of teaching. Dimensional modeling is a topic where spirited debates come up, some may even call it arguing. Warren didn’t argue, he listened to you and responded with subtle questions, he got you to realize your mistakes on your own.

Why do I keep referring to Warren in the past tense? He was diagnosed with a brain tumor shortly before joining my project. Regardless, he was the consummate professional, filled with enthusiasm and energy, and didn’t tell me until the end. The guy was fighting for his life, and we never knew it. Warren Thornthwaite passed away five years ago today.

Warren really loved data warehousing. He was immersed in it, and we bonded over our mutual obsession. I still miss him after all these years.

Feel free to share your experiences with him in the comments below.

Presenting Biml Frameworks Tonight at North Texas SSUG!

If you are in the DFW area, come down to the North Texas SQL Server User Group tonight at the Microsoft building in Irving, Tx. The meet and greet starts at 6:00pm and the presentation follows at 6:30pm. I’ll be giving the following brand new, deep dive presentation.

Creating Your Own ETL Automation Framework with Biml for Beginners

You may have heard of Business Intelligence Markup Language or even tried to learn it. But how can you take the time to learn it, while getting the rest of your work done. In this highly interactive presentation, I’ll guide you through the process of learning/using Biml, without sacrificing your daily productivity.
We’ll cover how to:

  • Identify “easy” ETL patterns to BimlScript
  • Take incremental steps in automating these processes
  • Move to metadata driven BimlScript
  • Tie it all together into a full-fledged framework

As a bonus, you’ll leave the presentation with a completed framework which dynamically generates most types of flat files that you can use in your own environment.

Anyway, additional details about the meeting can be found here. Hope to see some of you there!

Presentation Code Download

I'm Presenting Agile Data Warehouse Design with #Biml Tomorrow at SQL Saturday Houston!

This is just a quick note that I’ll be presenting an updated version of the following talk tomorrow at SQL Saturday Houston.

Agile Data Warehouse Design with Biml

Business Intelligence Markup Language is literally changing the way we think about Data Warehouse design and ETL. In this interactive presentation you’ll see how you can rapidly iterate through the Data Warehouse Development Cycle and provide ROI almost immediately. Learn how you can use BimlScript to automatically create staging tables, SSIS Packages, and other necessary objects during the modeling process.
Presentation Code Download

It’ll be a great time, and I can’t wait to see some of you there!

Presenting Azure Data Factory and SSIS Automation With Biml Tonight in Fort Worth!

If you are in the DFW area, come down to the Forth Worth SQL Server User Group tonight at the Frees and Nichols Headquarters. The meet and greet starts at 6:30pm and the presentation follows at 7pm. I’ll be giving an updated version of the following presentation.

Azure Data Factory and SSIS Automation With Biml

SSIS was originally designed to work in a world where all data was structured and on premise. As modern enterprise data projects increasingly move online, tools like Azure Data Factory (ADF) can extend SSIS capabilities to the cloud. Azure Data Factory is a flexible and powerful cloud based ETL service, but it doesn’t replace the functionality of SSIS.

In this session you will learn the advantages and disadvantages of both ETL platforms; how to use both in tandem to maximize productivity;
and how both ETL platforms, as well as any supporting structures, can be be automatically generated with Biml Script.

Anyway, additional details about the meeting can be found here. Hope to see some of you there!

I'm Presenting Tomorrow at SQL Saturday Dallas BI Edition!

This is just a quick note that I’ll be presenting an updated version of the following talk tomorrow at SQL Saturday Dallas.

Agile Data Warehouse Design with Biml

Business Intelligence Markup Language is literally changing the way we think about Data Warehouse design and ETL. In this interactive presentation you’ll see how you can rapidly iterate through the Data Warehouse Development Cycle and provide ROI almost immediately. Learn how you can use BimlScript to automatically create staging tables, SSIS Packages, and other necessary objects during the modeling process.
Presentation Code Download

Also, I’m bringing a bunch Biml t-shirts with me, courtesy of my friends at Varigence.

It’ll be a great time, and I can’t wait to see some of you there!

Presenting Azure Data Factory Automation with #Biml at the PASS Summit!

This is just a quick shout out for anyone attending the PASS Summit. I am giving two presentations, and couldn’t be more pumped. First, I’m giving the following Lightning Talk in 2 hours:

Three Things You Won’t Believe I Do With Biml

Room: 618-620
Time: 4:45PM – 6:00PM

You may have seen a basic Biml presentation in the past, perhaps a data staging pattern moving records from one database to another, and that’s all great. In this presentation I will amaze and astound you with advanced applications of Biml that I guarantee you have never seen. In front of your very eyes, I will use small amounts of BimlScript to tackle problems that have vexed ETL developers for years.

I will also be presenting the following 75 minute presentation on Friday.

Azure Data Factory and SSIS Automation With Biml

Room: 615-617
Time: Friday 3:30PM – 4:45PM

SSIS was originally designed to work in a world where all data was structured and on premise. As modern enterprise data projects increasingly move online, tools like Azure Data Factory (ADF) can extend SSIS capabilities to the cloud. Azure Data Factory is a flexible and powerful cloud based ETL service, but it doesn’t replace the functionality of SSIS.

In this session you will learn the advantages and disadvantages of both ETL platforms; how to use both in tandem to maximize productivity;
and how both ETL platforms, as well as any supporting structures, can be be automatically generated with Biml Script.

So, if you’re attending the PASS Summit, I’d love to see you there!

Presenting SSIS Performance Tuning With #Biml Tomorrow at #SQLSatOKC

Varigence_Football
Just a quick shout out that I’ll be at SQL Saturday Oklahoma City tomorrow to give an updated version of the following presentation.

SSIS Performance Tuning with BimlScript

You may heard that Biml can automate ETL loads, but what if your data warehouse is several terabytes with a tight load window. In this demo focused presentation, we’ll discuss how to use BimlScript to accomplish the following:
– Dynamically create multiple SISS packages for a data warehouse ETL load, each following a different best practice design pattern.
– Automatically execute each package multiple times and record the results for analysis.
– Determine which package version should be used in production.
The only way to be sure that your SSIS packages are performing optimally is to test them against multiple patterns. Make your life easier and let BimlScript do the work for you.
Presentation Code Download

Also, I’m bringing a big box of Biml schwag with me, courtesy of my friends at Varigence, including shirts, miniature footballs, laptop stickers, etc.

It’ll be a great time, and I can’t wait to see some of you there!

I'm Giving Two Biml Presentations at SQL Saturday Sioux Falls!

Varigence_Football
If you’re in the Sioux Falls, SD area this weekend, come see me present at SQL Saturday Sioux Falls #539! I’m going to fly up Friday afternoon, and will be giving the following presentations.

Agile Data Warehouse Design with Biml

Business Intelligence Markup Language is literally changing the way we think about Data Warehouse design and ETL. In this interactive presentation you’ll see how you can rapidly iterate through the Data Warehouse Development Cycle and provide ROI almost immediately. Learn how you can use BimlScript to automatically create staging tables, SSIS Packages, and other necessary objects during the modeling process.
Presentation Code Download

SSIS Performance Tuning with BimlScript

You may heard that Biml can automate ETL loads, but what if your data warehouse is several terabytes with a tight load window. In this demo focused presentation, we’ll discuss how to use BimlScript to accomplish the following:
– Dynamically create multiple SISS packages for a data warehouse ETL load, each following a different best practice design pattern.
– Automatically execute each package multiple times and record the results for analysis.
– Determine which package version should be used in production.
The only way to be sure that your SSIS packages are performing optimally is to test them against multiple patterns. Make your life easier and let BimlScript do the work for you.
Presentation Code Download

Anyway, it’ll be a great time, and I can’t wait to see some of you there!