Programming

pianoforte · Post by **pianoforte** » Wed Mar 28, 2012 1:25 am

Install Linux. Problems that need to be solved abound!

-From a proud Linux user since late 2010.

polar sky · Post by **polar sky** » Wed Mar 28, 2012 1:51 pm

Just programmed my clock radio the other day. It was a bitch, but after I read the manual I was setting my alarm and adjusting the brightness of the LED screen and everything. Thinking about giving tutorial classes at our local Senior Citizens Center.

Guido · Post by **Guido** » Sun Apr 01, 2012 4:36 pm

Ok, so I'm working full time on my emulator and it's getting better all the time. No DOS prompt yet, though, but close (I think).

Today I made a little debugging console for making debugging the whole thing easier:

http://pastie.org/private/9r6qa9smkh5ww2kslhza2w

Fun

.

Guido · Post by **Guido** » Sun Apr 01, 2012 11:50 pm

Here you can see it in action:

Screen positioning stuff isn't working yet so that is why the text looks garbled.

Cupz · Post by **Cupz** » Mon Apr 02, 2012 2:46 am

Guido · Post by **Guido** » Fri Apr 06, 2012 5:23 am

Now it is running the BIOS and the VGA BIOS correctly:

Waterbagel · Post by **Waterbagel** » Fri Apr 06, 2012 3:39 pm

i made my first program last night..

all it did was say "insert text here"

but i navigated my directories!
baby steps!

Pantheon · Post by **Pantheon** » Sat Apr 07, 2012 10:47 am

Guido, your work there is fantastic!!!

Guido · Post by **Guido** » Sat Apr 07, 2012 2:52 pm

Thanks! It's a lot of work (currently over 5700 lines of C++ and counting) but it pays off

. It currently involves long nights of assembly debugging (ie. I have a debugger built-in in my program to detect where the code execution goes wrong) and as always this can be quite tedious. Sometimes I spend a couple of hours in a row finding out where there is a problem through a maze of internal breakpoints and once I find it it usually takes like 1 minute to fix it.

I think I can almost boot MS-DOS now.. It already shows "Starting MS-DOS..." so I'm almost there

.

I plan to release the source code once it is all working.. I reckon this kind of stuff looks good on one's resume.

The Friendly Stranger · Tue Apr 17, 2012 7:42 pm

I'm not a programmer, but saw this, laughed and though you guys might identify with it.

Ender · Post by **Ender** » Wed Apr 18, 2012 1:31 pm

The Friendly Stranger wrote:I'm not a programmer, but saw this, laughed and though you guys might identify with it.

Hehe yup, that's pretty accurate. The only difference in my case is that I'm on the release team, which means I'm somewhere between a programmer, QA, and a release manager <_<. The biggest derp for me is when it's getting close to release time, and there's lots of tarball testing and such to do, and it takes a long time, and then another unplanned change ends up getting added to the branch and we have to do testing all over again.

Pantheon · Post by **Pantheon** » Fri Apr 20, 2012 11:57 am

I'm actually following in Guidos footsteps, but I've chosen the Thumb instruction set due to simplicity.
I've almost managed to disassemble most of the instructions, there are a few huge errors with operand decoding at the moment. No real point to all of this other than it a nice learning exercise, it could be used as part of a game scripting language, or as part of a game boy advance emulator.

http://pastebin.com/LKk8k8Nj

Guido · Post by **Guido** » Fri Apr 20, 2012 2:25 pm

Cool. Some advice: in the future you may want to refrain from using long blocks of switch/case statements and use an array of pointers pointing to function addresses with each element representing a different opcode. The gain in speed might seem negligible upon first inspection but it will become noticeable if you are going to execute millions of instructions per second. I also still use the former method, but once the CPU core is bug free I want to switch to using function lookup tables too. Actually I have two different classes now, one is the Disassembler class and the other one the CPU class. The main loop fetches the opcode that has to be executed next from memory, passes this on to the Disassembler class which fills a structure with the data it yields, and this structure is then given the to the CPU::execute functions, which like it says executes the instruction encoded in the struct and updates its internal state accordingly. I initially thought that this method would be good for the sake of clarity (and I still think it is), but I've noticed that this causes a massive speed bottleneck. I've benchmarked my emulator against the emulator of a friend who actually doesn't have an integrated disassembler but uses has a long list of switch/case statements and his emulator is MUCH faster than mine. What is even better than this method speed-wise is using a table-driven system like I said. Theoretically it is possible to cram all the processor data that you will need in your program into one or a couple of big tables, and then use these tables for function lookup, disassembly and even assembly if you need that. I think this is what the bigger x86 emulation projects such as Bochs do. QEMU uses an additional, rather ingenious method to maximise execution speed: opcodes are read from the stream, interpreted and subsequently dynamically recompiled into native code. So, if i'm not mistaken, the code that is to be emulated is disassembled and then reassembled into code that runs on the machine that QEMU is running on, and this code is then executed DIRECTLY (instead of truly emulated). This has the additional advantage of offering a framework that allows for, let's say, ARM emulation on an x86 system and vice versa. Quite advanced stuff, Fabrice Bellard is a genius.

Oh and in case anyone is interested, here is another (more recent) video of my emulator:

I'm currently writing an x86 emulator testing suite that makes certain groups of instructions (such as the rotate instructions, the shift instructions, the arithmetic instructions) operate on a 64kb block of random data, then compute a checksum of this block for comparison with other emulators. That should help eliminating any bugs that are left in my emulation of these instructions.

Pantheon · Post by **Pantheon** » Fri Apr 20, 2012 11:09 pm

Guido wrote: Cool. Some advice: in the future you may want to refrain from using long blocks of switch/case statements and use an array of pointers pointing to function addresses with each element representing a different opcode. The gain in speed might seem negligible upon first inspection but it will become noticeable if you are going to execute millions of instructions per second.

I was really banking on the fact that the compiler would turn my switch statement into some kind
of a jump table, I suppose I'd have to look at the disassembly to be sure. If it is performing successive if tests for each case then yes this would be a terrible method.

As for your idea of using a function pointer array, I have some issues with that. It has the potential to be a really clean solution, but I believe it would perform much worse than the switch statement.
Calling a function pointer stalls the entire CPU pipeline so that it cant plan ahead because its not yet sure what the next opcode will be before it reaches the call instruction (so i have read). Also, and it would depend on your function call type, pushing a new stack frame and all arguments would also add to the cost over the basic switch statement. You could get around that by using the declspec(__naked ) thing so that there is no stack frame created, and either making the arguments global or using registers. Also in order to get around the call instruction you could perhaps assemble your own jump table at runtime, using only jump instructions. You would have to be careful to jump back to a known location though. As I write this I'm starting to wonder if my facts are correct since I cant think why call would be slower than jump in terms of halting the pipeline...

Guido wrote: I also still use the former method, but once the CPU core is bug free I want to switch to using function lookup tables too. Actually I have two different classes now, one is the Disassembler class and the other one the CPU class. The main loop fetches the opcode that has to be executed next from memory, passes this on to the Disassembler class which fills a structure with the data it yields, and this structure is then given the to the CPU::execute functions, which like it says executes the instruction encoded in the struct and updates its internal state accordingly. I initially thought that this method would be good for the sake of clarity (and I still think it is), but I've noticed that this causes a massive speed bottleneck. I've benchmarked my emulator against the emulator of a friend who actually doesn't have an integrated disassembler but uses has a long list of switch/case statements and his emulator is MUCH faster than mine.

I can imagine that you are performing lots of data copying to and from the disassembly structure which would really bog things down. I think you are right, that would be the most clear method but also the slowest. My goal was much like your friends since I'm trying to work only from the packed 16bit instruction using masks and shifting.

Your task is way more complex then mine though, Thumb is quite a nice instruction set, but x86 still makes my brain hurt even thinking about it. This is to be my first real emulator so I'm trying to take baby steps here.

Guido wrote: Quite advanced stuff, Fabrice Bellard is a genius.

Yeah, I used his C compiler TCC in a VST project that could recompile itself at runtime. So I can reprogram it while it is running. The guy realy is amazing, I remember you showing me the link of his emulator writen in javascript running linux.

Guido wrote: Oh and in case anyone is interested, here is another (more recent) video of my emulator:

Very cool, it must be nice to see that familiar ram check running on your home brew emulator

Guido wrote: I'm currently writing an x86 emulator testing suite that makes certain groups of instructions (such as the rotate instructions, the shift instructions, the arithmetic instructions) operate on a 64kb block of random data, then compute a checksum of this block for comparison with other emulators. That should help eliminating any bugs that are left in my emulation of these instructions.

Thats a very nice idea for bug testing, I might have to steal that when I reach that stage

Ender · Post by **Ender** » Fri Apr 20, 2012 11:55 pm

I've been writing a little DCPU-16 compiler in Perl in my free time at work. So far it only understands the SET command.

Cupz · Post by **Cupz** » Sat Apr 21, 2012 11:54 am

No game programing here?

Currently working on a psychedelic horror game made out of flashing cubes and a text adventures in actionscript. The text adventure might be a bit too complicated.

Guido · Post by **Guido** » Sat Apr 21, 2012 2:02 pm

Pantheon, good point about the generation of a stack frame. I also hadn't thought of the fact that the compiler might generate jump tables for longer switch/case blocks. About the stalling pipeline, wouldn't this be true not only for CALLs but for every branch? Any conditional statement in a programming language, including switch/case blocks, generates branches I reckon.

Here is a screenshot of four emulators running my testing suite. Clockwise starting from the top left window: my emulator, QEMU, Bochs and Fake86. It's amazing that even mature emulators suchs as Bochs and QEMU apparently execute the arithmetic instructions slightly differently. But it's hard to get emulation completely right and results sometimes vary across different machines of the same architecture. For example, a friend in the possession of both an 486 system and a 8086 tried the rotate testing code on both these machines and the results were different. It is likely that there are some very very minor differences across these machines (I suspect that it has to do with the setting of the carry flag under specific conditions, in this respect I've noticed discrepancies between the Bochs source code and the Intel documentation of these instructions).

Pantheon · Post by **Pantheon** » Sat Apr 21, 2012 10:35 pm

Guido wrote:About the stalling pipeline, wouldn't this be true not only for CALLs but for every branch? Any conditional statement in a programming language, including switch/case blocks, generates branches I reckon.

Yeah I think your right. I was getting myself confused in that last post but yes all branches have the same effect on the pipeline. I think all the call function does over jump is just push the return address. I cant think of any way that is more efficient than a jump table in terms of performance.

I'm writing an ELF file loader at the moment so that I can run thumb programs directly from them. I also eventually want to program a tiny GDB server as part of my emulator so I can do some source level debugging. The GDB specifications however look pretty scary.

Pantheon · Post by **Pantheon** » Sun Apr 22, 2012 10:41 am

I think I have found a good reference for implementing a GDB server.
http://www.embecosm.com/appnotes/ean4/e ... sue-2.html

I wonder if it could be of any use to your emulator Guido? You do seem like you have a nice debugger built in already however.

Guido · Post by **Guido** » Tue Apr 24, 2012 5:23 pm

Using GDB for debugging is probably a more logical thing to do but it's more fun to roll my own ;D. I also never really liked GDB that much for some reason (I don't really like the AT&T assembly syntax for example..).

____twoism

Programming

Re: Programming