What is Virtual 8086, why is it causing problems for me?

Virtual 8086 mode is a sub-mode of protected mode. In short, virtual 8086 mode is whereby the cpu (in protected mode) is running a "Emulated" 16bit 'segmented' model (real mode) machine.

V8086 mode can cause all different kinds of problems for OS programmers.

The most common problem with v86 mode is that, you can't "enter" protected mode inside a v86 task.

That meaning, if you are running Windows or have emm386 in memory, you can't do a "raw" switch into protected mode (it causes an exception if I remember correctly!:)

There are a few other more "technical" problems people have when using v86 mode, mostly because v86 has some instructions "emulated" by what's known as a v86-monitor program, as the cpu is in protected mode, some instructions are high up on the security/protection level and running those directly would cause no-end of trouble for the OS.

Such technicalities are beyond the scope of a simple FAQ

 

How do I detect if my machine is in Virtual 8086 mode?

EFLAGS.VM is NEVER pushed onto the stack if the V86 task uses PUSHFD. You should check if CR0.PE=1 and then assume it's V86 if that bit is set.
detect_v86:
	smsw	ax
	and	eax,1		;CR0.PE bit
	ret