Azure bites

dkf

Java 1.1, because that still runs on recent JRE.

The only big problems you're going to have in that area relate to security components. Most code doesn't tinker with that stuff, but things that do will probably break.

Tsaukpaetra

@BernieTheBernie said in Azure bites:

@Tsaukpaetra said in Azure bites:

I suppose profiling a program outside the cloudz should be more efficient, but hey, free trial is free trial I suppose....

You should also consider that my machines at home are real vintage hardware. And yes, I know whom I tell that: you have a strong competitor, believe it or not!

You mentioned your PC had 4 cores with 8Gb of ram. We are not the same!

Tsaukpaetra

Status: There is no FUCKING way you've seen this password too many times before. Or ever!

Bulb

@Tsaukpaetra You might have hit a conflict in their password hash function or, even more likely, a false positive in the filter. Because while we have enough evidence that Microsoft is not immune to security blunders, I don't think they would be storing the passwords clear-text. So it will be some structure based on hashes, and probably a Bloom or cuckoo filter, which saves space, but allows for false positives.

Tsaukpaetra

@Bulb said in Azure bites:

cuckoo filter

Seems a bit ineffective at keeping out the bird shit everywhere IMO...

BernieTheBernie

@Tsaukpaetra For sure they have seen Fucking_fuckinG really often!

BernieTheBernie

I do not undersdtand Azure. Her behavior is so odd.
I wrote a "Function App" (equivalent to AWS Lambda) with a timer trigger. It needs a "secret" which I originally hard coded in the code. Later on, I created a "key vault", stored the secret there, and wrote a function to retrieve it. Wrapped in a try-catch, and in case of error use the hard-coded value and log an error - because getting that simple function app working was ... .
Anyways, the function app is now doing what it is expected to do. Well, almost. When I take a look at "application insights" and open the "failure" blade, I see that every invocation comes with one error. A 401 when accessing the keyvault.
Where the fuck does that happen? Because nowhere do I see my log output for that exception, it is something caught by Azure. But there it is.
So I decided to add another function to my function app for the purpose of testing and investigating. This time using an http trigger instead of a timer trigger because of chrontabic pain. Copied the function for accessing the keyvault, and added more logging.
And yes, it says that it could retrieve the secret from the keyvault. And under "Application Insights" - "Failures" there's my fancy 401:

And see: after the 401 which I do not see in the lines of my code, there is another access to the keyvault with a 200 which I see... I do not understand what's happening here (same keyvault, same secret).

Btw, I assumed a pronoun for Azure: she/her. Because

Azure is a woman.
:change-my-mind.meme:

HardwareGeek

@BernieTheBernie said in Azure bites:

Btw, I assumed a pronoun for Azure: she/her. Because
Azure is a woman.

Darn. I was going to make a joke about your use of her, but you anticipated and subverted it to so I'll have to settle for pointing out that she/her is not a pronoun; they're two pronouns.

dcon

@HardwareGeek said in Azure bites:

@BernieTheBernie said in Azure bites:

Btw, I assumed a pronoun for Azure: she/her. Because
Azure is a woman.

Darn. I was going to make a joke about your use of her, but you anticipated and subverted it to so I'll have to settle for pointing out that she/her is not a pronoun; they're two pronouns.

In today's world, I wouldn't assume that (anymore).

Tsaukpaetra

@BernieTheBernie said in Azure bites:

Btw, I assumed a pronoun for Azure: she/her. Because

Azure is a woman.
:change-my-mind.meme:

Of course Azure is a woman.

Arantor

On the contrary I thought Azure was a man. You give it requests, it'll get around to them when it feels like it and not necessarily promptly, but it will acknowledge them initially. And for bonus points, it will do everything it was asked, but badly, in the expectation and hope you won't ask again.

Tsaukpaetra

@Arantor I'll be generous in my characterization of the typical woman and not work to refute.

Unperverted Vixen

@BernieTheBernie That's all happening within Microsoft's SDK. After the service returns 401 Unauthorized, the SDK retrieves a new token from the appropriate place and then resubmits the request that failed.

You could argue that they should know they don't have a token yet and thus shouldn't even try, but since they need to do try-and-handle-a-401 for other cases like token expiration or permission changes, treating no-token-yet in the same way probably simplifies the code. For web apps it's not a performance issue, since the SecretClient is supposed to be a singleton. Not sure how that maps to functions, though.

Tsaukpaetra

@Unperverted-Vixen said in Azure bites:

Not sure how that maps to functions, though.

The context gets pretty quickly destroyed so I'd imagine it would happen somewhat often. 🤔

kazitor

@HardwareGeek said in Azure bites:

@BernieTheBernie said in Azure bites:

Btw, I assumed a pronoun for Azure: she/her. Because
Azure is a woman.

Darn. I was going to make a joke about your use of her, but you anticipated and subverted it to so I'll have to settle for pointing out that she/her is not a pronoun; they're ~~two pronouns~~a pronoun and a possessive.

MFW doesn’t know grammar:

HardwareGeek

@kazitor

She: personal pronoun (subjective case). (She did something.)
Her: personal (objective case) or possessive pronoun. (Help her. Personal pronoun acting as the direct object. Give her a thing. Personal pronoun acting as the indirect object. It's her thing. Possessive pronoun.)

Two pronouns.

kazitor

@HardwareGeek said in Azure bites:

It's her thing. Possessive pronoun.

I don’t know how confused you have to believe that ‘her’ is a pronoun here. Literally no noun group will fit in that context. Here’s a possessive pronoun: “It’s hers.”

Maybe someone was a bit confused to forget about indirect objects when ing , but you’ve helpfully been wronger now

Unperverted Vixen

@Tsaukpaetra The Azure Functions docs seem to say singletons persist past the lifetime of a single function execution...

Tsaukpaetra

@Unperverted-Vixen said in Azure bites:

The [...] docs

And you trust them?

Gustav

@Arantor said in Azure bites:

On the contrary I thought Azure was a man. You give it requests, it'll get around to them when it feels like it and not necessarily promptly, but it will acknowledge them initially. And for bonus points, it will do everything it was asked, but badly, in the expectation and hope you won't ask again.

It never tells you what's wrong and hopes you'll figure it out yourself. It nags you about unimportant stuff all the time and gets in the way of actual work. It shit talks other similar products behind their backs. And whenever you ask what's wrong, the only answer is "nothing, I'm fine".

Unperverted Vixen

@Tsaukpaetra Trust, but verify.

Arantor

@Gustav said in Azure bites:

@Arantor said in Azure bites:

On the contrary I thought Azure was a man. You give it requests, it'll get around to them when it feels like it and not necessarily promptly, but it will acknowledge them initially. And for bonus points, it will do everything it was asked, but badly, in the expectation and hope you won't ask again.

It never tells you what's wrong and hopes you'll figure it out yourself. It nags you about unimportant stuff all the time and gets in the way of actual work. It shit talks other similar products behind their backs. And whenever you ask what's wrong, the only answer is "nothing, I'm fine".

I know more men that do exactly this than women, you're not selling it to me as how this is an example of Azure being female.

Bulb

@BernieTheBernie said in Azure bites:

And see: after the 401 which I do not see in the lines of my code, there is another access to the keyvault with a 200 which I see... I do not understand what's happening here (same keyvault, same secret).

It is behaving as http endpoints requiring authorization are supposed to!

401 is not an error yet. It means authorization is required, so only then the http client library asks the token factory for an access token—which is obtained with the http://localhost:41023/msi/token/ request (I'd rather expect http://169.254.169.254/metadata/identity/oauth2/token, probably the function runtime has some wrapper for that)—and retries the original request with authorization, which then succeeds. And you only see the final, successful, response.

The , of course, is, that it shows up as a “failure” of the function, because it isn't.

@Unperverted-Vixen said in Azure bites:

You could argue that they should know they don't have a token yet and thus shouldn't even try, but since they need to do try-and-handle-a-401 for other cases like token expiration or permission changes, treating no-token-yet in the same way probably simplifies the code.

The HttpClient that does this handling is generic. It does not know whether it will need a token and, more importantly, it does not know what resource (audience) and scope it will need it for. It only learns these values—that it needs to pass to the TokenCredential.GetToken method—from the 401 response.

Unperverted Vixen

@Bulb said in Azure bites:

I'd rather expect http://169.254.169.254/metadata/identity/oauth2/token, probably the function runtime has some wrapper for that

That endpoint is specific for VMs. For app services (which should include function apps), the URL varies and needs to be extracted from the environment.

The HttpClient that does this handling is generic. It does not know whether it will need a token and, more importantly, it does not know what resource (audience) and scope it will need it for. It only learns these values—that it needs to pass to the TokenCredential.GetToken method—from the 401 response.

The handling isn't in HttpClient, though; it's in the Azure SDKs. HttpClient doesn't even know what a token is. It's just another HTTP header to it.

The Azure SDKs should already know the audience and scope beforehand. (Otherwise, it would never be able to do token caching.) I didn't think the 401 response even had any data, though I'll break Fiddler out tomorrow and verify...

Tsaukpaetra

@Unperverted-Vixen said in Azure bites:

I didn't think the 401 response even had any data, though I'll break Fiddler out tomorrow and verify...

It should tell you what kind of authorization it's expecting (for example digest or whatever). It is possible it may include more, but there's not mandatory stuff that needs to be there.

BernieTheBernie

@Unperverted-Vixen said in Azure bites:

@Bulb said in Azure bites:

I'd rather expect http://169.254.169.254/metadata/identity/oauth2/token, probably the function runtime has some wrapper for that

That endpoint is specific for VMs. For app services (which should include function apps), the URL varies and needs to be extracted from the environment.

The HttpClient that does this handling is generic. It does not know whether it will need a token and, more importantly, it does not know what resource (audience) and scope it will need it for. It only learns these values—that it needs to pass to the TokenCredential.GetToken method—from the 401 response.

The handling isn't in HttpClient, though; it's in the Azure SDKs. HttpClient doesn't even know what a token is. It's just another HTTP header to it.

The Azure SDKs should already know the audience and scope beforehand. (Otherwise, it would never be able to do token caching.) I didn't think the 401 response even had any data, though I'll break Fiddler out tomorrow and verify...

And anyway, there is the SecretClient which delegates to the HttpClient. And the SecretClient ought to know that some kind of Token is required, because otherwise it would be equal to retrieving a secret anonymously.
Which would be .
Or rather ?

Bulb

@Unperverted-Vixen said in Azure bites:

Otherwise, it would never be able to do token caching.

Does it even do it? The BlobClient or KeyVaultClient or whichever gets an instance of TokenCredential, and it might be different kind of TokenCredential for each client, so I suppose the best it can do is let you make the client static (and I suppose it is static if you use bindings (injection)).

@Unperverted-Vixen said in Azure bites:

I didn't think the 401 response even had any data, though I'll break Fiddler out tomorrow and verify...

Hm, I just tried requesting a blob with curl, and I'm getting a 404, not 401. On a URL that I copied straight from the portal, and I did check the storage account has access from all networks. And that is a resource that the front-end accesses in-browser with a SAS token.

BernieTheBernie

@Bulb said in Azure bites:

getting a 404, not 401

Yes, a BlobContainer set to private access does not reveal that you have an otherwise vaild URL. Because security.
I do not know how it would behave when you used an invalid SAS or whatever.
In the example I provided above, a KeyVault was used instead - perhaps there the test for access happens at the KeyVault level without a look at the specified secret (and a 404 only when you are allowed to access it, but the secret was not found?).

Unperverted Vixen

@Bulb said in Azure bites:

Does it even do it? The BlobClient or KeyVaultClient or whichever gets an instance of TokenCredential, and it might be different kind of TokenCredential for each client, so I suppose the best it can do is let you make the client static (and I suppose it is static if you use bindings (injection)).

It looks like both TokenCredential and SecretClient (and presumably the other similar clients) do some level of token caching.

Making a second request to the same key vault with the same SecretClient, even if for a different secret, sends a token on the initial request so there's no 401.

Making a request to a different key vault with the same SecretClient omits the token the first time, and thus the 401 is encountered. However, the authenticated attempt uses the same token as it did for the first key vault. (Meaning SecretClient saw the 401, asked TokenCredential for a token, and TokenCredential coughed up the exact same one as it had for key vault #1.)

Which, looking at the 401 like I promised, is understandable. The 401 body doesn't include anything informative, but the header does include both the ~~Azure AD~~Entra ID tenant to authenticate against as well as the resource to request a token for, and it's generic: https://vault.azure.net. So I was wrong about it not including those, but because it's generic, TokenCredential can safely cache it and reuse without needing to contact the token authentication endpoint again.

I'm guessing SecretClient must just be playing it uber-safe and not assuming that tokens will always use that generic resource.

boomzilla

@kazitor said in Azure bites:

@HardwareGeek said in Azure bites:

It's her thing. Possessive pronoun.

I don’t know how confused you have to believe that ‘her’ is a pronoun here. Literally no noun group will fit in that context. Here’s a possessive pronoun: “It’s hers.”

The proper name with a possessive fits it. "Her pronouns." / "Paula's pronouns."

Bulb

@Unperverted-Vixen said in Azure bites:

I'm guessing SecretClient must just be playing it uber-safe and not assuming that tokens will always use that generic resource.

Actually I think it is playing it safe and assuming the tenant will not always be the same, because that quite often indeed won't (managed identities don't work across tenants, but you may be using a client secret or client certificate for an app registrations and those do).

Bulb

@Bulb said in Azure bites:

managed identities don't work across tenants

That reminds me: Some time ago Microsoft introduced “federated credentials” where you can specify that an app registration (or a user-assigned managed identity) can be authenticated using a valid token with specific issuer, audience and claim. This can be used e.g. by Kubernetes pods to exchange the token that Kubernetes gives them for an Entra token to access Azure resources with. You could also use it to give workloads in Google or Amazon cloud to use their managed identities to get Azure managed identities.

However, they explicitly prohibit the issuer to be Entra, so you can't use it to give workloads in one Azure tenant managed identities in another Azure tenant. Unless you concoct your own service that will issue your tokens based on the Entra ones. That should work, because there are no requirements on the issuer, it just can't be Entra. Because beaver, I suppose.

Unperverted Vixen

@Bulb Ah, that makes sense. All my work has been within our one corporate tenant, so I didn't even think of that.

Bulb

@Unperverted-Vixen Yeah, I was contracting for this customer, who had, due to bad planning, the project spread across two different tenants, and then that customer had a customer who thought they want us to build something in their tenant, so I was juggling three at once.

And for this project we have the company tenant and a separate tenant for the production environment. It was me who said we should have a separate production tenant. My main reason is that the app registrations will be granted permissions by users, and likely in future into some customer tenants, and that means the app registrations can't be easily replaced with new ones in case we needed to move to another tenant e.g. if the product was sold. Currently the product has been carved out to a subsidiary; the users have not yet moved to a separate tenant, but eventually will and this way it won't affect the production tenant.

In general I'd say it is considered best practice to put projects you expose to customers to a tenant separate from the tenant with employee accounts and office subscriptions. But then if the project is self-contained, you won't be cross-referencing resources between those tenants. You mainly need to if your project interacts with customer resources, e.g. if you push data to customer storage—we don't do that yet, but have that option in the architecture.

BernieTheBernie

Since we do not do kink shaming here, I dare to admit:
I love Azure's Secret Magic

I created a KeyVault, and stored the connection string for a SQL Server there. Then I added an app setting named secretconnection to a function app (which was registered in Entra ID and granted the privileges to read secrets from a keyvault) with the value of @Microsoft.KeyVault(VaultName=bernieskeyvault;SecretName=berniessecret).
Now I can access it in the app with a simple Environment.GetEnvironmentVariable("secretconnection")

It worx. It retrieves the value stored in the keyvault (i.e. the clear text of berniessecret).
But only when the app is running on Azure. When I run it locally, the environment variable just gets resolved to @Microsoft.KeyVault(VaultName=bernieskeyvault;SecretName=berniessecret)....

Bulb

@BernieTheBernie The environment variable is just an environment variable, so it gets resolved to whatever you've set it to. The resolution happens in whatever starts the server that runs the app.

Gustav

@BernieTheBernie fuck you for not fully locking yourself into our ecosystem!

Bulb

@Gustav Every platform has a corresponding feature (with its own store) and there are some generic secret stores that you can install anywhere, so it does not really make much sense to use Azure keyvaults when you are not using them mostly with other Azure resources.

Bulb

@BernieTheBernie Also, the point isn't really convenience, the point is providing the secret to the application without giving access to the person deploying the application. This is achieved by granting the permission to read it to the app's managed identity.

If you run the function locally, it does not have managed identity, it can only use your identity (usually via the azure cli or powershell module). Which means you can access the secret anyway and can just resolve it by hand.

BernieTheBernie

@Bulb said in Azure bites:

The resolution happens in whatever starts the server that runs the app.

yes, Secret Magic
I know I can use a SecretClient to resolve it manually - and so before.

Bulb

@BernieTheBernie I don't like the way it's implemented as magic. In Kubernetes you explicitly configure environment variables with either value, which is simply a string, or a valueRef, which is an object that references a secret (kubernetes own secret storage)—it can also be three other types of an object referencing different things. That way it's clear at which layer the referencing happens and avoids using magic syntax. But otherwise it's the same—it's a feature of the service, not of the SDK.

BernieTheBernie

Another chrontabic pain. I have a "serverless" function which looks at a list of virtual machines, retrieves the current status of the machine, and "deallocates" it if it is not yet deallocated.
The trigger is [TimerTrigger("0 30 23 * * *")].
The invocation result is:

Bulb

@BernieTheBernie I vaguely recall there might have been some caveat about timers not working under some conditions … probably in the “consumption” plan, which is exactly what you'd be using in the free tier. to look it up, but it's shit either way.

Gustav

@BernieTheBernie looks like they dropped all pretenses of consistent and now it's just eventually.

BernieTheBernie

What a fun ! SQL Server is of course different from Microsoft SQL Azure (RTM) - 12.0.2000.8 Nov 2 2023 01:40:17 Copyright (C) 2022 Microsoft Corporation .
Because my ususal way of creating a backup just failed:

Msg 40510, Level 16, State 1, Line 1
Statement 'BACKUP DATABASE' is not supported in this version of SQL Server.

But of course, miles long powershell commands come to rescue:

Error: Statement 'BACKUP DATABASE' in not supported in this version of SQL Server

While writing this question on this site, many similar questions were suggested on the same topic by others, but I did not find the desired answer. I'm trying to write code in WPF C# to create a da...

Bulb

@BernieTheBernie said in Azure bites:

What a fun ! SQL Server is of course different from Microsoft SQL Azure (RTM) - 12.0.2000.8 Nov 2 2023 01:40:17 Copyright (C) 2022 Microsoft Corporation .

Ock Fourse it isn't. That's been mentioned a couple of times already.

Because my ususal way of creating a backup just failed:
Msg 40510, Level 16, State 1, Line 1
Statement 'BACKUP DATABASE' is not supported in this version of SQL Server.

That's kinda expected given you don't have access to a local disk on the server where the backup would normally go.

But of course, miles long powershell commands come to rescue:

Error: Statement 'BACKUP DATABASE' in not supported in this version of SQL Server

While writing this question on this site, many similar questions were suggested on the same topic by others, but I did not find the desired answer. I'm trying to write code in WPF C# to create a da...

You have a choice between:

The portal, if you are doing it manually anyway.
That powershell commandlet.
The CLI (az).
Calling the resource manager API with any other suitable REST API client.

Also if you use Entra (Azure AD) authentication to the database and the storage account, you can avoid passing all the secrets to the command; of course you'll still need the names of the database and the storage account, and because of the dumb way the resource manager operates, you have to give the resource group name even though the server name is already unique.

Bulb

@BernieTheBernie Oh, and for extra , the backups seem to be unreliable. Sometimes you run that command and the task hangs until your patience runs out, you abort it and run it again and it works. And sometimes, https://what.thedailywtf.com/topic/29043/sql-server-transaction-deadlocking-with-itself happens.

Bulb

This post is deleted!

BernieTheBernie

@Bulb said in Azure bites:

That's kinda expected given you don't have access to a local disk on the server where the backup would normally go.

I use that normally on my local machine, and my local SQL Server is configured not to be connectable from different machines. So never experienced what would happen on a remote sql connection.
Azure sql is "serverless"

I could do a "backup" with the help of SSMS: for tables and views, have it build create scripts which I copied into a text file. And for contents, I selected output to file for the queries...

BernieTheBernie

@Bulb said in Azure bites:

Sometimes you run that command and the task hangs until your patience runs out

The Azure SQL Server goes into sleep mode quite quickly. And then it takes about 55 seconds to wake it up...

Had to prepare for that with a a couple of retries in my äpps.