I have had that idea highlighted for me while I was setting up a KVM switch so that I can run both the beaglebone (Mill) and the 386 (Lathe) through the same monitor.
With everything plugged in I powered it all on and... It... didn't work. It would display the screen from the 386 but I could not change it to display from the beaglebone. I think what has happenend is the KVM switch is expecting to be plugged into PS/2 ports (That's why it has them I guess) but I am plugging it through an adapter into an old style keyboard connector and that's it. No mice or keyboard from the Beaglebone.
I think this is causing it to run out of power and just not work properly.
To fix this I am doing what I should have done in the first place and buying a cheapy from ebay that is completely passive. From the picture of it's guts I saw it is very basic which will be perfect for this as then there is less chance it will not work.
So sometimes it pays to just buy the right thing in the first place. But for a hobby, while it can be frustrating and demoralizing it usually isn't too bad. In a business situation however it can cause major problems.