Amazon Athena Source Control
-
I don't know how many people here use AWS, but as much as I love Athena, I'm pretty pissed about the lack of tooling for it. The AWS console is not great. I've got it sort of working in DataGrip, but it can't tell the difference between tables and views.
I am interested in source controlling the views, and I'm too lazy to copy and paste them every time. I think there may be something I can write in python, but
Why hasn't anyone done this before? Are people not versioning their Athena views or do my google skills just failing me?
-
Since such thing apparently doesn't exist.
@apapadimoulis business idea. A non shitty client for athena that also does source control.
-
@dangeRuss said in Amazon Athena Source Control:
I love Athena
I'd probably fuck her too, but I've never heard of her!
-
@dangeRuss what's Athena all about?
-
@apapadimoulis war.
-
@apapadimoulis said in Amazon Athena Source Control:
@dangeRuss what's Athena all about?
Interactive SQL - Amazon Athena - AWS
Amazon Athena is a serverless, interactive analytics service that provides a simplified and flexible way to analyze petabytes of data where it lives.
It doesn't solve my problems (which involve massive local data generation, not data already in S3) so I've not investigated further.
-
@apapadimoulis said in Amazon Athena Source Control:
@dangeRuss what's Athena all about?
Basically you can have a ton of data sitting out on s3, in varying formats, like tsv, parquet, etc and you can run queries on it, with joins and all the good stuff, and you only get charged for the amount of data read.
There are some limitations, but mostly its pretty great. The only problem is lack of decent tooling. There is a jdbc driver and probably odbc as well, but in datagrip, for example, it doesn't recognize what is a view vs table. And there is no support for source control.
Athena is basically presto underneath.
-
@dangeRuss interesting, how does one end up with all those different types of data? Are these aggregated from third parties or something?
-
@apapadimoulis said in Amazon Athena Source Control:
@dangeRuss interesting, how does one end up with all those different types of data? Are these aggregated from third parties or something?
Yes it could be third party data, combined with internal data, etc. The thing that used to be called Big Data and is now just called data.
-
@dangeRuss said in Amazon Athena Source Control:
it doesn't recognize what is a view vs table
If you're just querying it, why do you care? (Updates would be something else, but mapping them back to the datasources would be crazy-hard so I expect them to be not supported.)
-
@dkf said in Amazon Athena Source Control:
@dangeRuss said in Amazon Athena Source Control:
it doesn't recognize what is a view vs table
If you're just querying it, why do you care? (Updates would be something else, but mapping them back to the datasources would be crazy-hard so I expect them to be not supported.)
I would like to edit it sometimes. Maybe even version control it.
