Azure bites

So we are trying out Azure Load Testing. My team lead created a simple test just to make sure it worked. Well, it ended up making 600,00 requests, and it looks like it cost $13. Not a huge amount of money, but he was expecting such a small first test to be free, so we don't want to proceed until we're sure we're in control of the process.

So by default, it's set by virtual users, and this was the one that gave us 600K hits. So I tried setting it to the other option, Requests Per Second, because that looks like it gives you more direct control. But it gets complicated, and my test ended up having 300K requests, which was a lot more than I was aiming for.

So I figured that before I actually ran it, I would go back and confirm that it was set up the way I wanted it. But this is not easy to tell. In particular, there's a difference between how much information you can see if you go back and double-check.

Screen you see if you used the default, Virtual Users: (Note that I redacted the test URL, in this picture, it is actually filled out.)

With virtual users redacted.png

But if you select Requests Per Second and go back to see what you set it to:

with requests per second.png

Note that a lot of information is missing in the second screen. So you can't tell how it's actually set.

I may also cross-post this in the Help topic, because, in addition to it being a WTF, I'm looking for advice.

Bulb

@Sampa said in Azure bites:

I'm looking for advice.

The portal is sometimes buggy, but you should be able to see the underlying JSON somewhere, and that should be authoritative source of information—on the summary page of every resource there should be a JSON View linklet in the top left, and there is usually an “Export Template” at the bottom of the left side menu where it should include subresources.

Kamil Podlesak

@Arantor said in Azure bites:

One assumes that if you are standing on the hose, no water is forthcoming and inspiration and enlightenment are as surely blocked as the hose.

Since the expression is not recorded before 20th century (more like 1930s onwards), it is almost certainly based on the 1895 movie: https://moviessilently.com/2020/12/21/the-sprinkler-sprinkled-1895-a-silent-film-review/

Bulb

The az account get-access-token, which is used for most authentication needs during development and deployment, is dumb.

You can be logged in to multiple tenants and subscriptions at the same time, with the same account (that might be a guest in other tenants) or multiple accounts.

One of the accounts is set as active, but you can use the other accounts by explicitly providing the subscription ID.

But if you provide only tenant ID—which is all most applications know—rather than looking for some entry that matches that tenant, it tries to get token for the active account in the specified tenant. So if you have one account that is a guest in many places, it works, but if you actually have separate accounts, it will give you an error and you have to set the active account.

… a bit of a pain when you are switching between two projects with separate accounts.

Bulb

So I'm reconfiguring this Azure database when terraform spits out:

│ Status: "OperationInterrupted"
│ Code: ""
│ Message: "The operation on the resource could not be completed because it was interrupted by another operation on the same resource."
│ Activity Id: ""
│ 
│ ---
│ 
│ API Response:
│ 
│ ----[start]----
│ {"name":"bflmpsvz-bflm-psvz-bflm-psvzbflmpsvz","status":"Failed","startTime":"2024-06-18T13:03:33.613Z","error":{"code":"OperationInterrupted","message":"The operation on the resource could not be completed because it was interrupted by another operation on the same resource."}}
│ -----[end]-----

There was nothing else going on with that resource, I'm still setting it up for crying out loud!

Before that I was rebuilding AKS when it died of some other spurious error getting a token. This whole API thing is a house of cards. A particularly rickety one.

Of course it completed fine on re-running. That's why I'm reluctant trying to put this thing into CD and stick with the commit+apply+push-if-successful approach. It would be too much back-and-forth to run it on build server.

Bulb

https://learn.microsoft.com/en-us/azure/azure-sql/database/metrics-diagnostic-telemetry-logging-streaming-export-configure?view=azuresql&tabs=azure-cli said:

Azure CLI version 1 is antedilluvial. I didn't even know such thing existed until now. The version 2 has been current for years. And it's a pretty thin layer over the REST API too. Adding the support should be simple. There's really no excuse for not supporting it.

… though, it actually might be supported and the documentation is out of date instead. That's always a possibility with Microsloth.

BernieTheBernie

Yesterday evening, I decided to change a "secret" stored in the keyvault. Actually, not really a secret, just an email address where to send a report to, because I want to learn while playing around with azure. 3 function äpps receive that "secret".
After changing it around 7:30 pm, I tested it. And it failed.
Um, ok, eventual consistency. Try an hour later.
Failed again.
Um, ok, sometimes, it may take a little longer.
The äpp which run at 11:30 pm still used the old "secret".
The äpp running in the morning at 7 am did not send a report. Neither to the old email address nor to the new address. At least, no email arrived.
The äpp running in the afternoon at 5:30 pm used the new email address.
Wow, 22 hours to get a secret in the keyvault updated.
After all, a is just some uplifted , and in thick fog you should reduce your speed because safety.

izzion

@BernieTheBernie said in Azure bites:

Yesterday evening, I decided to change a "secret" stored in the keyvault. Actually, not really a secret, just an email address where to send a report to, because I want to learn while playing around with azure. 3 function äpps receive that "secret".
After changing it around 7:30 pm, I tested it. And it failed.
Um, ok, eventual consistency. Try an hour later.
Failed again.
Um, ok, sometimes, it may take a little longer.
The äpp which run at 11:30 pm still used the old "secret".
The äpp running in the morning at 7 am did not send a report. Neither to the old email address nor to the new address. At least, no email arrived.
The äpp running in the afternoon at 5:30 pm used the new email address.
Wow, 22 hours to get a secret in the keyvault updated.
After all, a is just some uplifted , and in thick fog you should reduce your speed because safety.

If only someone had published a manual explaining the expected behavior and how to cachebust.

TwelveBaud

@izzion MSDN cheesemoving in three... two... one...

HardwareGeek

@izzion said in Azure bites:

If only someone had published a manual

P(MS has published a manual with useful info) = 1 - P(Finding said manual)

izzion

@HardwareGeek said in Azure bites:

@izzion said in Azure bites:

If only someone had published a manual

P(MS has published a manual with useful info) = 1 - P(Finding said manual)

Thankfully, we now have Google Powered By AI to help us root through the cruft

BernieTheBernie

@izzion said in Azure bites:

@BernieTheBernie said in Azure bites:

Yesterday evening, I decided to change a "secret" stored in the keyvault. Actually, not really a secret, just an email address where to send a report to, because I want to learn while playing around with azure. 3 function äpps receive that "secret".
After changing it around 7:30 pm, I tested it. And it failed.
Um, ok, eventual consistency. Try an hour later.
Failed again.
Um, ok, sometimes, it may take a little longer.
The äpp which run at 11:30 pm still used the old "secret".
The äpp running in the morning at 7 am did not send a report. Neither to the old email address nor to the new address. At least, no email arrived.
The äpp running in the afternoon at 5:30 pm used the new email address.
Wow, 22 hours to get a secret in the keyvault updated.
After all, a is just some uplifted , and in thick fog you should reduce your speed because safety.

If only someone had published a manual explaining the expected behavior and how to cachebust.

That's very practical and nondisruptive when you have dozens of äpps which are scaled out to many instances each in production, and now you have to perform a pseudo configuration change in order to restart all the äpps.
And of course, the new pseudo configuration change setting will linger around from now on eternally, and with the next change another one gets added, and later on some poor code hog will have to try to find out what those settings mean...

Bulb

@BernieTheBernie That's why you typically have two keys or secrets for each thing. So you can rotate the other one than you are currently using and then switch to it.

Or you just do the “pseudo” configuration change by Terraform (or Pulumi) automatically (unfortunately the switching between primary and secondary key is somewhat difficult to do with Terraform, but propagating the current version of the secret everywhere is easy). You really want to use Terraform (or Pulumi) with this cloud thing anyway, other paths lead to insanity.

Or you avoid needing secrets in the first place. All the Azure resources should be possible to be connected with IAM using managed identities now.

Unperverted Vixen

@BernieTheBernie said in Azure bites:

That's very practical and nondisruptive when you have dozens of äpps which are scaled out to many instances each in production, and now you have to perform a pseudo configuration change in order to restart all the äpps.

If your apps need to always have the latest secret, then they should retrieve it directly from the key vault on use.