Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Unity Catalog] - Upgrade task of uploading Date/Time Dims to a notebook that creates them (deploy bash script) #984

Open
3 tasks done
thesqlpro opened this issue Dec 19, 2024 · 2 comments · May be fixed by #995
Open
3 tasks done
Assignees
Labels

Comments

@thesqlpro
Copy link
Contributor

thesqlpro commented Dec 19, 2024

Dynamically create Date/Time Dimensions into tables using a notebook (potentially the setup.py notebook) and remove the need to upload these files into DLS and load them into tables. They should go directly into the Unity Catalog.
Task for #765

DoD

  • Date Dimension dynamically created and loaded in Setup.py notebook
  • Time Dimension dynamically created and loaded in Setup.py notebook
  • XXDate/Time Dimension file upload to ADLS path removed (commented out in deploy_intrastructure.sh) - See comment belowXX
@ydaponte
Copy link
Collaborator

@thesqlpro, please do add a DoD and fill in the metadata.

@thesqlpro thesqlpro self-assigned this Dec 23, 2024
@thesqlpro thesqlpro added the P1 High Priority label Dec 23, 2024
@thesqlpro thesqlpro added P0 Uber URGENT priority and removed P1 High Priority labels Dec 23, 2024
@thesqlpro
Copy link
Contributor Author

This cannot be done yet as the SQL Dedicated pool uses these raw CSVs for it's external tables (date and time dims)

Date/Time Dimension file upload to ADLS path removed (commented out in deploy_intrastructure.sh)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment