Mustafa (my manager) knew about my current fascination with learning the _real_ basics of computers (electronics, kernel, compilers, etc.), so he arranged for me to attend the GCC workshop held by IIT Bombay this week. My first reaction was that I would be wasting my time there since I didn't know the first thing about compiler theory and was a novice at best in assembly language programming. He said it wouldn't hurt to try. So I had to try.
I got a chance on the day before the workshop to read up some about things like IRs, RTLs, etc. It was enough that I would not be completely lost from day 1. But I did not attend day 1 at all, thanks to the countrywide strike that crippled all public transport (and some people too). So I spent that day working and also trying to cover what would otherwise have been covered in day 1 -- the various phases of compilation, passes, gray box probing to find out some more about intermediate outputs, etc. I got hold of last year's slides, so it was a little easier.
So I finally made it to days 2, 3 and 4. The first thing I noticed during the lecture sessions was that the professor really knew his stuff. He was well acquainted with the internal layout of gcc and was able to explain it well enough that I really _got_ it. Overall I come out of the sessions today with much more knowledge about gcc than I could ever gain in 4 days on my own. Here are some observations that I made during the course of this session.
The professor really knew his stuff. I say this again so that it does not look like I am ignoring that. There are also a lot of really talented individuals at the GRC, who are doing some pretty interesting research based on gcc. The trouble though is that there seem to have been no efforts whatsoever to share these ideas upstream.
One such idea is the Generic Data Flow Analyzer (GDFA). It is a patch to gcc that provides a data flow analyzer, which can be used to find and eliminate dead code or unused variables. It adds a gimple pass to the compilation sequence and intends to replace the current dead code elimination and unused variable elimination passes with the same code called with different parameters. While the idea is pretty interesting, the sad thing is that there are no signs of an attempt to push this idea upstream. All I could find was an announcement to the gcc mailing list, but no request for comments or for inclusion of the patch.
This is only one of many more ideas that are brewing in the GRC in the minds of some very talented people. But one felt that these ideas were being used only to get degrees and nothing was being done to actually test their feasibility in live production level code. It would be nice to see some of these ideas actually presented upstream with a genuine interest in getting them incorporated.
To conclude, it is a pretty good session for those who want to get started with learning compilers and gcc internals.
Philip had recently posted on libyahoo2-users that Yahoo is planning to open its instant messaging platform API to the public. It has been delayed a bit since then, but it is surely due.
So how does this change things for libyahoo2 or any other FOSS implementations? For one, the fun of digging through binary data and trying to make sense of it will be gone ;) But on a serious note, we can hope to have some more consistency in behaviour and support will definitely improve. I'm not very keen about the fact that the support will be over HTTP, but I guess it works well for them. For now we can only wait for their announcement before we know what the entire thing looks like. If it is anything like the messages that the current official yahoo! messengers send, then it's only really a wrapper around their old pain of a protocol. But this does not really use JSON, so it is likely that they're writing a fresh implementation. In any case, there is still time for it and in that time, we have some decent work going on on the libyahoo2 code base.
In other news, Kai Zhang has been working on implementing chat room support for libyahoo2 as his Fedora Summer Coding project. His code can be found here. Other than the brief comments in the git logs, everything seems to be quite ok. A bulk of the feature set is already in, so that is pretty good progress. Once the entire feature set is completed and tested, I will have them included in the main libyahoo2 source tree. Following that will be a release and a rebase on Fedora. This will be a good rebase compared to the ugly one the last time around, where I broke all API compatibility in an effort to revamp the authentication support.
So I started dabbling with assembly language programming a couple of days ago. This was the next logical step in the "going lower down" move I have been doing ever since I started writing programs in Visual Basic some years ago (there, I admitted it). Since then I went through C#, Java, C++, C and now finally assembly. And it is fun to watch a program die in so many innovative ways. It is helping me understand the internals of a program much better.
One of the first things I learnt about assembly programming was that I needed to use completely different syscall numbers and instructions for x86_64 as compared to i386. For example, the syscall number for exit on i386 is 1 while on x86_64 it is 60. Same goes for write -- 4 on i386 and 1 on x86_64. I spent half an our trying to figure out why my program was calling fstat on x86_64 while a similar program built with --32 would work fine.
Crossing all these hurdles, I finally wrote a slightly more complicated (but still useless) program than a hello world. This is a program that takes in an integer string through the command line, converts it to an integer, converts it back to string and prints it back out. Pretty useful huh :)
Now for the interesting part in the code. I always thought of dynamic memory allocation as something you can only do through the OS using the brk() and/or mmap() syscalls. Generally we do this indirectly through malloc() and friends. But what I ended up doing in my program is allocating memory on the stack on the fly. Here's the code snippet:
movb $0x0a, (%rsp) decq %rsp next_digit: movq $0, %rdx divq %rdi addq $0x30, %rdx # Hack since we cannot 'push' a byte movb %dl, (%rsp) decq %rsp
The complete code along with the makefile is at the end of this post. You can build it if you have an x86_64 installation. What I do above is simply:
- Read a digit from the number
- Move the stack pointer ahead to make room for a byte
- Store the ascii representation of that number into that byte
I could not use the push instruction itself, since it can only push 16, 32 or 64 bit stuff on to the stack (with pushw, pushl, pushq). If you push a single byte value, it will be stored in one of the above sizes, not in just 1 byte. What I wanted was to create a string on the fly without limiting myself to a fixed size array, so this seemed to be the only approach. While this works, I still need to find out a few more things about this:
- Is it safe?
- If it is safe, then is there a similar way to do this in C without embedding assembly code? This would be really cool, especially in usage scenarios such as the above. Admitted that the above scenario is pretty useless in itself, but I'm sure there must be similar examples out there that are at least a little more useful.
The code:
.section .data usage: .ascii "Usage: printnum-64 <the number>\n" usagelen = . - usage .section .text .globl _start # Convert a string representation of an integer into an int .type _get_num, @function _get_num: push %rbp movq %rsp, %rbp movq 0x10(%rbp), %rdx mov $0x0, %rcx mov $0x0, %rax nextchar: # Iterate through the string movb (%rdx), %cl cmp $0x0, %rcx je call_done subq $0x30, %rcx imulq $0xa, %rax addq %rcx, %rax incq %rdx jmp nextchar # Convert a number into a printable string .type _print_num, @function _print_num: push %rbp movq %rsp, %rbp movq 0x10(%rbp), %rax movq $0x0a, %rdi # Hack since we cannot 'push' a byte and increment # %rsp by only 1. push will push whatever it has as # a 16, 32 or 64 bit value (pushw, pushl, pushq) movb $0x0a, (%rsp) decq %rsp next_digit: movq $0, %rdx divq %rdi addq $0x30, %rdx # Hack since we cannot 'push' a byte movb %dl, (%rsp) decq %rsp cmp $0x0, %rax jne next_digit movq %rsp, %rbx addq $0x1, %rbx movq %rbp, %rcx subq %rsp, %rcx push %rcx push %rbx push $0x01 call _write jmp call_done # Wrap around the write system call .type _write, @function _write: push %rbp movq %rsp, %rbp movq 0x10(%rbp), %rdi movq 0x18(%rbp), %rsi movq 0x20(%rbp), %rdx movq $0x01, %rax syscall jmp call_done # I always do this when I am done with a function call call_done: movq %rbp, %rsp pop %rbp ret #Program Entry point _start: # Command line arguments: # The parameter list is: # argc: The number of arguments # argv: The addresses of all arguments one after the other # They can be popped out one by one pop %rax cmp $0x2, %rax jne error # Pop out the first arg since it is the program name, but # keep the second so that it can be fed into the next function pop %rax call _get_num push %rax call _print_num jmp exit error: push $usagelen push $usage push $0x2 call _write movq $0xff, %rax exit: movq %rax, %rdi movq $60, %rax syscall
The makefile:
32: as --32 $(target).s -o $(target).o ld -melf_i386 $(target).o -o $(target) 64: as $(target).s -o $(target).o ld $(target).o -o $(target)
If you save the source as foo.s, you can build it with:
make target=foo 64
The past week has been quite hectic, with a lot of juggling between different things I have been wanting to do. So here's what I had on my mind:
- I have been looking to learn more about compilers. I goofed off in college and missed out on the same course that was taught twice. I always understood enough to fool my teachers into thinking I knew it all, but not enough to really know it all. Or some for that matter. So now I want to make up for it.
- I had not touched ayttm and libyahoo2 for quite a while. So I wanted to do something there
- Kushal had asked me if I could package libraw for Fedora because some random app needed it. He asked me because I knew autotools and I could autotoolize the project before I package it.
- Rahul pointed out this cool little command line audio player called gst123. I had been looking to write something like this for some time now but I just could not wrap my head around gstreamer. I tried it and immediately fell in love. I just had to package it for Fedora.
- Work at my day job. Lots and lots of work.
- Work at my day job. Lots and lots of work. Yes, it is worth mentioning twice
And so here's what I actually ended up doing over the week:
- I had bought 3 books to study compilers. They're just lying there since I haven't had enough time to actually start studying.
- Nothing on ayttm and libyahoo2. Not enough time
- Packaged libraw and submitted for a package review. There is no activity on that bug report yet, but there was some action before it. Libraw upstream does not like autotools, so I had to hand-write a configure script to detect stuff. I also looked up and tried out the app for which Kushal wanted me to package libraw. The app is Shotwell, a photo management program. And it is good; I'm starting to use it for my photographs now. I'm glad I decided to package libraw for it.
I packaged gst123. The package has been approved and I have already submitted an update for F-13. I did this while on a bus from Pune to Mumbai :D
gst123 is a really cool app, try it out. It might not play internet radio streams right out of the box (my use case), but you can easily pipe/grep/cut your way to getting it to work. Here's how I play the radio stream from Absolute Radio:
gst123 `curl -s http://network.absoluteradio.co.uk/core/audio/ogg/live.pls?service=vcbb | grep File1 | cut -d '=' -f 2`
See, it's so easy!
Oh yeah, work at my day job. Lots and lots of work.
The FAD (Fedora Activity Day) was announced over a month ago with an intention to get some real work done during an event. I really only had a chance to participate in 1/4th of the FAD (1/2 day on Saturday), since I had to fly to Bangalore on Saturday evening to spend the weekend (or whatever was left of it) with family. But that was enough to get whatever I wanted out of the event.
Being pretty much a newcomer into the Fedora community, there wasn't much that I could think of to directly contribute but I wanted to do something. I really only maintain 1 package, which also does not have much traffic, so I wasn't exactly brimming with ideas. Rahul helped me there by asking me to do an Autotools workshop. I was also looking forward to meeting some of the guys I had met at FOSS.in last year; Susmit, Hiemanshu and Sayamindu. I could not meet Hiemanshu (did he come at all?), but it was good to meet Susmit and Sayamindu after quite a long time.
We started the day with my autotools workshop; I hope at least someone found it useful. I demonstrated the process of autotoolizing a simple C program using the same example I used during my Fedora classroom session earlier this month: linkc. The main reason I keep choosing this program is that I am too lazy to find or write anything on my own. The other reason is that the program helps to cover quite a few things at one go -- it is small, it has an external dependency, a subdirectory and some distributable files. So all those things win over the fact that the app just doesn't work as advertised. Oh well...
Once the only "session" of the day was over, everyone announced their aims for the two days while Sankarshan distributed some swag (t-shirts, stickers and buttons). After that it was pretty much everyone working on their own stuff. Me too.
Only a couple of days before FAD, Ray van Dolson added me as a co-maintainer for libyahoo2 in Fedora so that we could share the workload of doing releases/bug fixes. After discussion with him, I decided to do a libyahoo2 release into rawhide during the event. So I finally had something that I could do, which was much closer to Fedora.
I knew that the release would break freehoo, a console messenger for yahoo since libyahoo2 1.0.0 broke all backward compatibility, so I set about fixing that. The result was a bug report with a patch to fix freehoo to build with the latest libyahoo2. Finally, I also changed ayttm to dynamically link against libyahoo2 instead of cloning the code base all the time. There was absolutely no incentive in maintaining two code bases for it, so it finally had to go.
By the time the ayttm change was done, it was time to leave. But before that, Kushal asked me to take a look at libraw to see if I could pitch in with something there. So I will be looking at autotoolizing it and packaging it for Fedora. I was supposed to do it today, but all of my day was spent in playing catch-up with work at my day job. Maybe I'll have more time tomorrow for it.
