Onyx' adventures in virtualization

Onyx

Okay, I'm sick of this shit. I want my Linux. And I want my games. And I'm too lazy to fuck around with Wine (and some games just don't run regardless).

So, I'm trying the PCI-e passthrough thing again. This is the plan:

Run Win10 through QEmu with my Debian testing (what is that these days, buster?) install as a host
Pass my Nvidia 1060 to Windows and just use the builtin Intel APU for Linux. It will serve the regular daily use just fine and I intend on running games in Windows anyway
Use Looking Glass to actually see my Windows screen, thus eliminating the need for a separate screen just for the Windows VM.

Let's see how this goes. I'm about to reboot to switch to Intel APU as primary now (hopefully just changing it to primary in UEFI settings is enough, but fuck knows these days). I also still have my dual boot going and will be testing performance in games to see how much I lost if this works. I'm planning on testing in Doom (the 2016 one, naturally) in OpenGL and Vulcan, and I'll get something DirectX based that's a power hog going as well (likely Space Engineers, there's a benchmark map someone made floating around on the Steam workshop).

Well, reboot, do some stuff unrelated to this experiment in the mean time and I'll be back. Let the rage commence!

PleegWat

@Onyx said in Onyx' adventures in virtualization:

Debian testing (what is that these days, buster?)

As I understand debian versioning, that means you're doing the testing. The alternative is tested, which means other people have been given plenty of time to test and as a result what you get is ancient.

Onyx

@PleegWat said in Onyx' adventures in virtualization:

@Onyx said in Onyx' adventures in virtualization:

Debian testing (what is that these days, buster?)

As I understand debian versioning, that means you're doing the testing. The alternative is tested, which means other people have been given plenty of time to test and as a result what you get is ancient.

I meant the release name. Current stable is stretch. (Debian and their silly "naming releases after Toy Story characters" schtick).

Anyway, immediately, disaster strikes: the freaking Intel card has no DisplayPort outputs, and I'm all outta spare video cables of all kinds. Great. I'll make due, somehow.

The other thing is that it seemingly still loads the Nvidia driver. Everything runs like molasses. Think Windows XP without graphics drivers level. Time to purge Nvidia drivers and see if that does a thing. I hope it's not UEFI fucking with me as well, though.

Dreikin

@Onyx said in Onyx' adventures in virtualization:

@PleegWat said in Onyx' adventures in virtualization:

@Onyx said in Onyx' adventures in virtualization:

Debian testing (what is that these days, buster?)

As I understand debian versioning, that means you're doing the testing. The alternative is tested, which means other people have been given plenty of time to test and as a result what you get is ancient.

I meant the release name. Current stable is stretch. (Debian and their silly "naming releases after Toy Story characters" schtick).

Anyway, immediately, disaster strikes: the freaking Intel card has no DisplayPort outputs, and I'm all outta spare video cables of all kinds. Great. I'll make due, somehow.

The other thing is that it seemingly still loads the Nvidia driver. Everything runs like molasses. Think Windows XP without graphics drivers level. Time to purge Nvidia drivers and see if that does a thing. I hope it's not UEFI fucking with me as well, though.

Onyx

Okay, two reboots later and we're fine. First was because Nvidia's binary drivers love to hook themselves into the kernel in idiotic ways so I didn't trust that just restarting X would work, and the second one because the actual issue was me adding a nomodeset switch to boot options before, since this machine refuses to boot into Linux without that because, once again, Nvidia.

Time to actually start reading the instructions and getting the VM going. As soon as I remember where I downloaded the Windows 10 ISO to...

mott555

@Onyx said in Onyx' adventures in virtualization:

Pass my Nvidia 1060 to Windows

I think NVIDIA drivers intentionally block PCI passthrough unless you have a workstation card (Quadro). I have never tried though.

@Onyx said in Onyx' adventures in virtualization:

Use Looking Glass to actually see my Windows screen, thus eliminating the need for a separate screen just for the Windows VM.

That looks like a cool utility.

Onyx

@mott555 said in Onyx' adventures in virtualization:

I think NVIDIA drivers intentionally block PCI passthrough unless you have a workstation card (Quadro). I have never tried though.

There's jiggery-pokery for that. Apparently. I'm hoping that's what I was missing the last time and now it works.

Onyx

Two reboots later, because is and idiot and he forgot to run update-grub after adding the intel_iommu=on to kernel commandline options:

onyx@SHODAN:~$ sudo dmesg | grep -e DMAR -e IOMMU
[sudo] password for onyx: 
[    0.000000] ACPI: DMAR 0x00000000ABE22708 0000B8 (v01 INTEL  BDW      00000001 INTL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.000000] DMAR: Host address width 39
[    0.000000] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.000000] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.000000] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.000000] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.000000] DMAR: RMRR base: 0x000000ade8b000 end: 0x000000ade99fff
[    0.000000] DMAR: RMRR base: 0x000000af000000 end: 0x000000cf1fffff
[    0.000000] DMAR-IR: IOAPIC id 8 under DRHD base  0xfed91000 IOMMU 1
[    0.000000] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.000000] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.000000] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.000000] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.458609] DMAR: No ATSR found
[    0.458662] DMAR: dmar0: Using Queued invalidation
[    0.458701] DMAR: dmar1: Using Queued invalidation
[    0.458779] DMAR: Setting RMRR:
[    0.458839] DMAR: Setting identity map for device 0000:00:02.0 [0xaf000000 - 0xcf1fffff]
[    0.461451] DMAR: Setting identity map for device 0000:00:14.0 [0xade8b000 - 0xade99fff]
[    0.461537] DMAR: Setting identity map for device 0000:00:1a.0 [0xade8b000 - 0xade99fff]
[    0.461621] DMAR: Setting identity map for device 0000:00:1d.0 [0xade8b000 - 0xade99fff]
[    0.461683] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.461739] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.461911] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    0.713282] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    0.713316] AMD IOMMUv2 functionality not available on this system
[    7.738646] [drm] DMAR active, disabling use of stolen memory

We has IOMMU!

Stealing a script from Arch wiki to check stuff aaaand...

onyx@SHODAN:~$ ./iommu_check 
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06)
IOMMU Group 10 00:1c.3 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 [8086:8c96] (rev d0)
IOMMU Group 11 00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6]
IOMMU Group 12 00:1f.0 ISA bridge [0601]: Intel Corporation 9 Series Chipset Family Z97 LPC Controller [8086:8cc4]
IOMMU Group 12 00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82]
IOMMU Group 12 00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus Controller [8086:8ca2]
IOMMU Group 13 03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 0c)
IOMMU Group 14 04:00.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 41)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] [10de:1c02] (rev a1)
IOMMU Group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
IOMMU Group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06)
IOMMU Group 3 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06)
IOMMU Group 4 00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1]
IOMMU Group 5 00:16.0 Communication controller [0780]: Intel Corporation 9 Series Chipset Family ME Interface #1 [8086:8cba]
IOMMU Group 6 00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad]
IOMMU Group 7 00:1b.0 Audio device [0403]: Intel Corporation 9 Series Chipset Family HD Audio Controller [8086:8ca0]
IOMMU Group 8 00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [8086:8c90] (rev d0)
IOMMU Group 9 00:1c.2 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 [8086:8c94] (rev d0)

Yup, I can see both video and audio from the 1060 in the list. Cool.

Well... Time to give this a go:

0_1534166983436_d285a852-f49e-4e36-84d4-f9d09ecd5d80-image.png

EDIT: Aaaand I've done a boo-boo by giving Windows only a 40 GB drive, I wasn't paying attention when clicking... I guess that'll install though, so I can at least test all the settings and shit and then just replace the image file... ugh, fiiiiineeee.

blakeyrat

@Onyx Even if this works, I bet it'll have like 58 problems every day. Just use Windows in the first place.

Onyx

@blakeyrat said in Onyx' adventures in virtualization:

@Onyx Even if this works, I bet it'll have like 58 problems every day. Just use Windows in the first place.

I have about as many problems of a different nature with Windows installed "proper" every day anyway. It balances out.

Onyx

Cortana just scared the shit out of me, sitting in a backgrounded window since I only have one screen at the moment.

Who thought that was a good idea?

EDIT:

0_1534168036093_b436efc8-07c6-429c-a923-1ce6a19a4af7-image.png

Fuck security questions. Have a Red Dwarf reference.

EDIT2: THREE security questions? REALLY? Jesus fuck.

blakeyrat

@Onyx said in Onyx' adventures in virtualization:

I have about as many problems of a different nature with Windows installed "proper" every day anyway. It balances out.

If you don't tinker with shit, you don't have problems.

Onyx

Well, there we go, this is where the fun starts:

0_1534171502639_e6793caa-3765-4835-a128-5d37701193c3-image.png

Trying to figure out what is wrong here is... interesting, at the very least. It's the thing where the instructions are made for someone who can barely open a GUI, and any troubleshooting threads are seemingly filled with experts on both sides. Being in the middle sucks.

I'm going to see what I can dig up and see how long I'll even play with it today further, given the other stuff I want to do.

blakeyrat

@Onyx Well obviously group 1 isn't viable, duh.

anonymous234

@Onyx Protip: Windows, unlike Linux, lets you NOT set a password for your account in the first place.

Onyx

@Onyx said in Onyx' adventures in virtualization:

I'm going to see what I can dig up

Well, there's my problem:

onyx@SHODAN:~$ ./iommu_check | grep "Group 1 "
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] [10de:1c02] (rev a1)
IOMMU Group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)

My PCI-e host is in the same group as the GPU, and all devices in the group need to be bound to VFIO at the same time.

So now I'm stuck being unclear on things:

can I just pass it through? I have no other cards in there, and it does seem that it's only PCI-E 16x controller that's the problem, not PCI-E in general
if I pass it through, will it break shit?
can I grab a Debian kernel with a patch that makes this shit work? Or will I have to compile from source? Because I'm no too keen on the latter.

More at 10...

Onyx

Ok, so likely the last update for today, I'm back on my regular Windows install because I want to play some games now...

Anyway, I was being an idiot, once again, and I only added the HDMI audio device to the VM, not the GPU itself as well. Derp.

So, seems to start but it either hardlocks my machine, or redirects all the input and output to the VM which... doesn't show the image on the screen. Yay. I think it's the latter though, judging by disc activity I can see. Will probably get an SSH connection to it going and try to poke around as the VM is trying to start tomorrow or later tonight, if I decide to mess with it in the evening.

So yeah, maybe I'll have a working thing at some point? Fingers crossed.

mott555

@Onyx Are you using a desktop PC? Or is this a laptop with the Optimus dual-GPU setup? If it's the latter, I wouldn't expect it to work, the two devices already used a shared framebuffer (NVIDIA card writes all its output into the Intel GPU's framebuffer and only the Intel GPU is wired up to video outputs IIRC) and I'd expect setting one up for PCI passthrough to not really work.

TimeBandit

@blakeyrat said in Onyx' adventures in virtualization:

If you don't tinker with shit, you don't have problems.

Yeah, don't disable your Firewall or you can't install fonts

Onyx

@mott555 Nah, it's not a muxless setup, it's a desktop. I seem to be on the right track though.

Captain

This shouldn't be so hard. Why qEmu over Virtualbox?

Onyx

@Captain I'm pretty sure PCI passthrough isn't available there. At least not as far as I know. Is it? AFAIK it only works if it's using the builtin KVM hypervisor, not whatever VirtualBox uses.

Also, setting up the VM was no pain, it's the passthrough that's the tricky bit.

Polygeekery

@Onyx said in Onyx' adventures in virtualization:

Cortana just scared the shit out of me, sitting in a backgrounded window since I only have one screen at the moment.
Who thought that was a good idea?

Complete fucking idiots. Only a complete fucking idiot would think that having your computer talk to you during setup was a good idea.

Polygeekery

@blakeyrat said in Onyx' adventures in virtualization:

@Onyx said in Onyx' adventures in virtualization:

I have about as many problems of a different nature with Windows installed "proper" every day anyway. It balances out.

If you don't tinker with shit, you don't have problems.

Riiiiiiiiight.

Polygeekery

@Onyx have you considered just running two separate installs and swapping hard drives when you want to game? I used to do that back in the day with a hot-swap drive bay. It is a hell of a lot simpler and more reliable than what you are attempting to do.

TimeBandit

@Polygeekery said in Onyx' adventures in virtualization:

It is a hell of a lot simpler and more reliable than what you are attempting to do.

Dual-boot is simpler

Polygeekery

@TimeBandit said in Onyx' adventures in virtualization:

@Polygeekery said in Onyx' adventures in virtualization:

It is a hell of a lot simpler and more reliable than what you are attempting to do.

Dual-boot is simpler

But it is not less problematic.

Maybe it has all gotten better over the years but I never had any luck getting Windows and Linux to play nice together in dual-boot. Swapping drives is the only thing I ever had that worked until I started using VMWare Workstation.

TimeBandit

@Polygeekery said in Onyx' adventures in virtualization:

Maybe it has all gotten better over the years but I never had any luck getting Windows and Linux to play nice together in dual-boot.

As long as you don't re-install Windows, you're fine. Because Windows will overwrite your bootloader without asking.
Why? Because, fuck you, that's why

I guess your issue was a long time ago

Polygeekery

@TimeBandit said in Onyx' adventures in virtualization:

@Polygeekery said in Onyx' adventures in virtualization:

Maybe it has all gotten better over the years but I never had any luck getting Windows and Linux to play nice together in dual-boot.

As long as you don't re-install Windows, you're fine. Because Windows will overwrite your bootloader without asking.
Why? Because, fuck you, that's why

I guess your issue was a long time ago

I specifically remember one issue that was a long time ago (XP days) where a Windows update screwed up the bootloader.

Onyx

I'm dual-booting successfully right now. I just get annoyed by rebooting. So, trying this. If it doesn't work, it doesn't work. I won't be too broken up about it

Luhmann

@Onyx said in Onyx' adventures in virtualization:

jiggery-pokery

Must be a @Polygeekery alt

Atazhaia

Seeing as the PCIe controller is in the same group as the actual card, that's bad. You cannot pass through the controller, only the card. So you're pretty much screwed unless you can change the devices in the group to only contain the two devices that make up the actual card.

I have got PCI Passthrough to work and I have learnt a few things from it. First of all, use a HEDT PC as they are more likely to actually have hardware support for it. Using it on a mainstream PC is very hit-and-miss, as either the support is missing or the grouping is bad. Although it's hit-and-miss regardless of motherboard, because it's not exactly a well-used feature.

Also, use a Radeon as the card to pass through because Nvidia are dicks about passing through gaming GPUs.

And depending on how much you want to optimize, you'd need plenty RAM, dedicated USB controller and disk controller that can be passed through, and tracking down the core layout for your CPU to pin the virtual cores/threads to the physical cores/threads.

Also, while fun to poke at it's also plenty headache-inducing to get working in a stable state. So my suggestion: Dualboot! It's a lot easier to setup a working dualboot setup. Sure, it's also a bit of extra work due to Windows being made of pure incompetence, but once it's up and running it's good. At least I haven't had any issues with my setup.

Tsaukpaetra

@Onyx said in Onyx' adventures in virtualization:

security questions

Huh, that must be new. Home version, I assume?

Either that or it assumed that blank password would never need security questions to reset.

anonymous234

@TimeBandit said in Onyx' adventures in virtualization:

@blakeyrat said in Onyx' adventures in virtualization:

If you don't tinker with shit, you don't have problems.

Yeah, don't disable your Firewall or you can't install fonts

This but unironically.

I'm pretty obsessive in keeping all my settings to the default unless absolutely necessary, because the moment I touch anything anywhere, stuff WILL break.

mott555

It's really too bad VirtualBox's 3D acceleration features don't really work.

sloosecannon

@Tsaukpaetra said in Onyx' adventures in virtualization:

@Onyx said in Onyx' adventures in virtualization:

security questions

Huh, that must be new. Home version, I assume?

Either that or it assumed that blank password would never need security questions to reset.

He's probably making a Microsoft account

Onyx

@sloosecannon said in Onyx' adventures in virtualization:

@Tsaukpaetra said in Onyx' adventures in virtualization:

@Onyx said in Onyx' adventures in virtualization:

security questions

Huh, that must be new. Home version, I assume?

Either that or it assumed that blank password would never need security questions to reset.

He's probably making a Microsoft account

Nope, pro and local account. Downloaded a fresh ISO from MS the other day. Don't know what to tell you, didn't get those on previous installs either.

As for the whole "just dual boot" and "Nvidia sucks for this" - I know all that. It's still something I want to try and get working. This is for my own machine and my own personal use, if shit doesn't work or it breaks, fine, I'm not crashing someone's production server. But it's something I really want to try and do.

JBert

Back when IOMMU and VTd were new I thought you pretty much had to use the Xen hypervisor so that basically all operating systems have to go through the hypervisor rather than one giving up its stake and letting you pass hardware through to a VM.

Have you looked at it and found it unwieldy, or could it be that my knowledge is just years out of date?

Tsaukpaetra

@sloosecannon said in Onyx' adventures in virtualization:

@Tsaukpaetra said in Onyx' adventures in virtualization:

@Onyx said in Onyx' adventures in virtualization:

security questions

Huh, that must be new. Home version, I assume?

Either that or it assumed that blank password would never need security questions to reset.

He's probably making a Microsoft account

No, from the screenshot:

0_1534204331060_07681d3f-5104-431f-b4e3-cb31d3b0cbe4-image.png

pie_flavor

@blakeyrat said in Onyx' adventures in virtualization:

@Onyx Even if this works, I bet it'll have like 58 problems every day. Just use Windows in the first place.

pie_flavor

My roommate managed to virtualize Windows off his Macbook with a graphics card connected via USB. So it's definitely doable.

Tsaukpaetra

@pie_flavor said in Onyx' adventures in virtualization:

a graphics card connected via USB

Man do I hope it was at least USB 3.0...

pie_flavor

@Tsaukpaetra No, USB-C. It wouldn't be nearly fast enough otherwise.

Unperverted Vixen

@pie_flavor That sounds more like Thunderbolt 3, rather than USB 3.x. Same USB-C cable, different protocol.

Zecc

@Polygeekery said in Onyx' adventures in virtualization:

Only a complete fucking idiot would think that having your computer talk to you during setup was a good idea.

Or a blind person, I guess.

topspin

@TimeBandit said in Onyx' adventures in virtualization:

@Polygeekery said in Onyx' adventures in virtualization:

Maybe it has all gotten better over the years but I never had any luck getting Windows and Linux to play nice together in dual-boot.

As long as you don't re-install Windows, you're fine. Because Windows will overwrite your bootloader without asking.
Why? Because, fuck you, that's why

It will also fail to install Language Packs if it can't ~~overwrite~~recognize the bootloader. Because, fuck you, that's why.
Yes, I guess technically, there's like one line of text in the bootloader the language pack might update, too.

Onyx

@Zecc said in Onyx' adventures in virtualization:

@Polygeekery said in Onyx' adventures in virtualization:

Only a complete fucking idiot would think that having your computer talk to you during setup was a good idea.

Or a blind person, I guess.

I discarded that theory because everything else before that is not doable for a blind person, this is just the very end of the setup.

Unless new PCs with Windows preinstalled start you off there now? I don't know, the last computer I bought with Windows preinstalled had Windows ME on it, so...

Zecc

@Onyx said in Onyx' adventures in virtualization:

Unless new PCs with Windows preinstalled start you off there now?

Happened to me. I was very happy for volume hardware buttons, because it seems they also cater to the deaf.

HI I'M CORTANA —

Bye Cortana. Please don't come back.

Onyx

Oh, ye of little faith:

0_1534256012448_d2ded243-254f-404f-a79d-6be9e4ea46a2-image.png

Excuse the potato quality, light sucks in here right now, curtains are on as tightly as possible.

TimeBandit

@Onyx

From now on, only the monitor on the right will reboot sporadically?