Azure -- is there an equivalent to Amazon S3?
-
Specifically, I need these features:
- Ability to serve files over HTTPS reliably
- Ability to give the files reasonable URLs (
our-storage-server.account-23256.azuresites.com/file.gifor something similar) - Ability to see full server logs for the files sent-- the full requested URL including query string specifically (the intent is to use these logs to track web analytics data, the way a lot of analytics packages do with S3-hosted filse)
I've made an Azure Storage Account, and put a couple files in it, but:
- The serving from a Storage Account seems to be unreliable-- I've frequently had Azure return errors instead of the file requested*, which leads me to believe Storage Accounts are not intended for the purpose of hosting content
- To make them "public" (able to be served over HTTPS to the Internet-at-large) Azure gives the files a ridiculously huge URL with query string already added (this would make parsing the logs of accesses to those files very annoying)
- I can't find any way of accessing the logs of the Storage Account logs in any case
I find it very difficult to believe Azure has nothing that competes directly with Amazon S3-- am I just missing it in their confusing portal?
*) The error I get is:
<Error> <Code>InvalidHeaderValue</Code> <Message> The value for one of the HTTP headers is not in the correct format. RequestId:15a0714a-001a-0037-3578-2d93f7000000 Time:2017-09-14T16:45:44.7552797Z </Message> <HeaderName>x-ms-version</HeaderName> <HeaderValue/> </Error>From Googling around it seems to just be a "it just happens some times, shrug" type of error.
-
I found this article: https://discourse.snowplowanalytics.com/t/porting-snowplow-to-microsoft-azure/1178
Looks like there's an Azure feature called Event Hubs I should be looking into?
-
This is WAY too fucking complicated.
All I need is static file hosting with the ability to download the server's log files. That's all I need. Why is this so hard?
-
@blakeyrat What sort of storage are you trying to use right now? We use the Blob Storage, and it works pretty well for us.
-
Read the first post please.
All I want to do is host a single static file at a reasonable URL (and HTTPS), then be able to view the weblogs for it. EDIT: if you're asking what storage the logs should be kept in, then the answer is I do not care, I just need to be able to get at them.
https://www.microsoft.com/middleeast/azureboxes/cloud-hosting-for-a-static-website.aspx
This Azure tutorial might be more relevant, but it doesn't mention being able to look at the site's logs at all.
-
Ah, I think I finally found what I need:
Let me set up an SFTP login and see if I get the logs I'm expecting.
-
@blakeyrat said in Azure -- is there an equivalent to Amazon S3?:
Read the first post please.
I did, and this
@blakeyrat said in Azure -- is there an equivalent to Amazon S3?:
To make them "public" (able to be served over HTTPS to the Internet-at-large) Azure gives the files a ridiculously huge URL with query string already added
doesn't describe the URLs you get with Blob Storage, which look more like
hence why I asked what type of storage you were using.
-
@raceprouk said in Azure -- is there an equivalent to Amazon S3?:
doesn't describe the URLs you get with Blob Storage, which look more like
Well ok; so that meets one of the three requirements. What about the other two?
Your question was confusing because I'm not trying to use any kind of storage. I'm trying to put a file on a web server.
In any case, I figured it out on my own.
Turns out you have to create a "Web App" (Azure has a very different definition of the word "application" than I do), FTP to it to upload your pixel.gif file, then go into the settings and turn on web server logging which is off by default because... reason.
Web Apps give you simple URLs (xxx.azurewebsites.net) give you HTTPS "for free" and let you turn on standard IIS logging. And seem to work fine even if you have only static files and nothing any sane person would call an "application".
-
Blakeyrat - Hopefully this might clear some things up (for the future, I see you have solved your problem)...
A storage account is more akin to a raw disk drive [no filesystem/partitions] than to something you interact with directly. Additionally the operations you do perform directly are optimized for usage within the Azure environment....
So WebApp is appropriate for serving things as web type content [accessed via http:/https:].
Another choice (probably not a good one for your recent situation) is a Virtual Machine which would have VHD and these would be kept on the Storage Account. Here would could install IIS or use an other means (and it does NOT have to be a windows OS) that you would use for accessing files on a computer.
There are many others.
Yeah, the Azure Portal SUCKS! [specifically the Ibiza version] but Azure is really much more capable of doing complex things than Amazon and this leaks through so the simple is not as somple as one would like...
-
@thecpuwizard said in Azure -- is there an equivalent to Amazon S3?:
the Ibiza version
they definitly took the wrong pill on that one ...
-
@thecpuwizard The real problem is the answer to the question in the thread's title is "no". There's no equivalent to Amazon S3.
You can simulate Amazon S3 by making a Web App and ignoring 95% of the features it has and probably pay more $$$ for it (although not a factor for me, since work's paying for it), but anything like S3? Nope.
-
@blakeyrat "This is WAY too fucking complicated" describes my entire (admittedly small) experience with Azure.
It's like they built a fantastic product for trained engineers, but forgot to make it usable for everyone else.
-
I did a trial and within my trial period my Linux box got turned into a zombie spam server.
Now I didn't have a particularly complicated root password. But also everything was open by default. I think on amazon you specifically have to expose ports to the interwebs.
-
@blakeyrat said in Azure -- is there an equivalent to Amazon S3?:
Azure return errors instead of the file requested
Oh great, I hope not. Our game has a level that instructs clients to download up to 2k files from the blob storage when they join, and though it's not critical that all these files get downloaded, it would desync their experience from the rest of the clients....
-
-
@dangeruss said in Azure -- is there an equivalent to Amazon S3?:
particularly complicated root password.
@dangeruss said in Azure -- is there an equivalent to Amazon S3?:
But also everything was open by default.
Could you imagine a firewall that defaulted to everything blocked? Nobody would be able to use any VM they created and would have to go through the hard way to drill all the way down to their (probably automatically generated) Network Security Group to add ports to allow!
-
@anonymous234 The worst thing is that the UI keeps changing, so tutorials are always about things as they were when they were written.
There are a lot of good things in there, and some of them are even easy, but how you're expected to figure it out is beyond me.
-
@dangeruss said in Azure -- is there an equivalent to Amazon S3?:
I did a trial and within my trial period my Linux box got turned into a zombie spam server.
Now I didn't have a particularly complicated root password. But also everything was open by default. I think on amazon you specifically have to expose ports to the interwebs.
Huh?
When I set up a forum testbed on Azure a few weeks ago, I wasn't able to connect to it until I found the firewall and opened port 80. Everything was locked down tight, just like on AWS.
-
@blakeyrat said in Azure -- is there an equivalent to Amazon S3?:
@thecpuwizard The real problem is the answer to the question in the thread's title is "no". There's no equivalent to Amazon S3.
Largely depends on what one means by "equivalent"....
You can simulate Amazon S3 by making a Web App and ignoring 95% of the features it has and probably pay more $$$ for it (although not a factor for me, since work's paying for it), but anything like S3? Nope.
When using S3, the most recommend approach is to couple with either and AWS Storage Gateway or the newer Gateway Stored Volumes. Both of these also cost more money.
The pricing models are different but the end result is usually (80%+ of the time) close enough that it is fairly easy to pick a solution for either that beats the price of the other without any prima facie differences (obviously there are differences, and so which one is "better" requires pretty careful analysys.
Pretty sure you don't care, but I use Azure about 85%, AWS about 10%, and remaining choices about 5%. For clients it is about even at 45% for each AWS and Azure and 10% for the remaining. In almost all of those cases, the preference really was made based on "passion" rather than a pure mathematical and comprehensive analysis.
-
Static Website Hosting In Azure Storage | Build5Nines
Traditional shared hosting providers generally cost anywhere between $8 - $10 USD per month. The reason is you need to reserve some CPU and Memory resources
Enable and manage Azure Storage Analytics logs (classic)
Learn how to monitor a storage account in Azure by using Azure Storage Analytics.
Those might be helpful, though if you want detailed access logs then a low-tier web app with Application Insights hooked to it could be better.
If it's just serving content, I'd go for a blob storage.
-
@maciejasjmj said in Azure -- is there an equivalent to Amazon S3?:
If it's just serving content, I'd go for a blob storage.
Can you get the web logs?
Seriously, guys, the top post has THREE things I need. Maybe you can get the web logs with blob storage, but damned if I could figure out how.
-
Use Storage Analytics to log details about Azure Storage requests. See what requests are logged, how logs are stored, how to enable Storage logging, and more.
Storage Analytics logging records details for both successful and failed requests for your storage account.
Does that cover it?
-
@maciejasjmj Yeah it looks like the info I actually need would be in RequestURL column.
But whatever, I've finished it now. You posted too late by like 3 hours.
-
@magus said in Azure -- is there an equivalent to Amazon S3?:
There are a lot of good things in there, and some of them are even easy, but how you're expected to figure it out is beyond me.
That sounds like all things that MS does on the Web.
-
The two posts above seem to confirm two points.
- Information is hard to find, and that is a problem. Microsoft definitely suffers from this, but the problem is wide spread.
- Just because one can not find something does not be is does not exist.
-
@thecpuwizard said in Azure -- is there an equivalent to Amazon S3?:
Just because one can not find something does not be is does not exist.
If I can't find it, and don't otherwise know that it exists, the state of existence of a thing is of great unimportance to me.
-
@dkf said in Azure -- is there an equivalent to Amazon S3?:
If I can't find it, and don't otherwise know that it exists, the state of existence of a thing is of great unimportance to me.
My post was in response to a prior comment of "...there is nothing...", which is about the existence and totally unrelated to the importance (or lack thereof to any specific person.
As far as your statement (quoted above), there at many things in life that one does not know of the existence, and yet the importance is quite high. Just consider the people at Edgewood's military base between 1955 and 1975...
-
@thecpuwizard said in Azure -- is there an equivalent to Amazon S3?:
@dkf said in Azure -- is there an equivalent to Amazon S3?:
If I can't find it, and don't otherwise know that it exists, the state of existence of a thing is of great unimportance to me.
My post was in response to a prior comment of "...there is nothing...", which is about the existence and totally unrelated to the importance (or lack thereof to any specific person.
You dropped this: )
-
@ben_lubar said in Azure -- is there an equivalent to Amazon S3?:
@thecpuwizard said in Azure -- is there an equivalent to Amazon S3?:
@dkf said in Azure -- is there an equivalent to Amazon S3?:
If I can't find it, and don't otherwise know that it exists, the state of existence of a thing is of great unimportance to me.
My post was in response to a prior comment of "...there is nothing...", which is about the existence and totally unrelated to the importance (or lack thereof to any specific person.
You dropped this: )
Your HtmlCleaner plugin automatically closes tags, so I don't see a problem.
-
@thecpuwizard said in Azure -- is there an equivalent to Amazon S3?:
it is fairly easy to pick a solution for either that beats the price of the other without any prima facie differences
What? Like, whichever solution you choose you can go to the other platform and find something similar but cheaper? Why not switch back to the first one and find an even cheaper one then? Repeat enough and you've got your solution for free
-
@jaloopa said in Azure -- is there an equivalent to Amazon S3?:
What? Like, whichever solution you choose you can go to the other platform and find something similar but cheaper? Why not switch back to the first one and find an even cheaper one then? Repeat enough and you've got your solution for free
Similar is not same.... So there cand be a "similar but less than" at each swap....Eventually you pay nothing for nothing - and are 100% vendor independent.
-
I need more helps:
- Is the filesystem for Azure web sites encrypted-at-rest by default? (The filesystem where it stored IIS logs specifically)
- Can anybody locate documentation from Microsoft that clearly states that it is?
- If not, is there something in the Azure portal I can toggle to encrypt that data at-rest, then screenshot to prove it's toggled on?
I'm 99.9% sure that Azure does encryption-at-rest of all data it stores, but I can only find documents referring to Storage Accounts, not the storage that's associated with Web Applications.
-
@blakeyrat said in Azure -- is there an equivalent to Amazon S3?:
If not, is there something in the Azure portal I can toggle to encrypt that data at-rest, then screenshot to prove it's toggled on?
I scoured our app services and there's nothing there for it.
But, as counter, I'd ask for proof that you can get at the data at rest to begin with, because that's really the only reason you need it (and, as far as I know, it's rather difficult to get azure data in an at-rest state). But that's neither here nor there.
According to this document
This article provides an overview of Azure Data Encryption at-rest, the overall capabilities, and general considerations.
Microsoft is committed to providing encryption at rest options across cloud services and to provide customers suitable manageability of encryption keys and access to logs showing when encryption keys are used. Additionally, Microsoft is working towards the goal of making all customer data encrypted at rest by default.
It's "In progress", but until you see that "Encryption" tab (like you do in Storage), assume it's not there.
But, if you're running your own VM, and that disk is stored on a storage container that is encrypted, then yes, by default it's encrypted at rest by definition and virtue of being stored in the encrypted container.
