Hell Oh Entropy!

Life, Code and everything in between

Fedora Activity Day at Pune: Towards a more secure Fedora

Posted: Nov 02, 2014, 07:10

Huzaifa had wanted to do a Security FAD in Pune for a while to tackle the really high number of open security bugs in Fedora. We had initially set a date for September but we pushed it forward since Huzaifa was not available. In the end, Huzaifa was not not available even on the rescheduled date, so PJP took over ownership of the event.

I wasn’t expecting a lot of people to attend given the nature of the activity and as it turned out, there were 14 signups with 7 showing up finally. We also had a few people joining remotely, which was awesome. We also had a Docker event running in parallel at the venue (the Red Hat Pune office), so we had more company at lunch.

Everyone barring PJP came in on India Standard Time, i.e. late by a few minutes to an hour or so. We started a bit late as a result, with a quick introduction to security in Fedora by PJP. After the talk and questions we didn’t waste any time and quickly got down to triaging security bugs. Our plan of action was to take ownership (by setting fst_owner= in the bugzilla whiteboard) of security bugs we understand and start working on driving them to conclusion. What this implied was that we would have to follow up after the FAD to ensure that the bugs were closed.

I started from the oldest bugs (dating back to 2011!) and managed to own 8 bugs by the end of the day. We had many a spirited discussion over what constituted a security bug (most of us understood OS security to a fair extent, but were not security experts) and my impression was that all of us went home a bit wiser. I learned that xen is a horrible horrible package - it bundles a bazillion projects into itself, due to which fixing flaws in the original project is not sufficient and xen would need to be checked and fixed separately.

Overall we had a pretty good day where 36 bugs got new owners - we managed to reduce the total backlog (of unowned bugs) from 370 to 334. Hopefully some of us will continue to work in our spare time (I know I’ll try) and bring that backlog down further.


Blank lines in /etc/netgroup

Posted: Jan 27, 2014, 11:36

While doing a code audit, I noticed that netgroups queries resulted in warnings in valgrind on my system:

==14597== Invalid read of size 1
==14597==    at 0xBB735E0: _nss_files_setnetgrent (files-netgrp.c:106)
==14597==    by 0x4F4954E: __internal_setnetgrent_reuse (getnetgrent_r.c:139)
==14597==    by 0x4F49879: setnetgrent (getnetgrent_r.c:181)
==14597==    by 0x4033EA: netgroup_keys (getent.c:493)
==14597==    by 0x402370: main (getent.c:1011)
==14597==  Address 0x51fe63f is 1 bytes before a block of size 120 alloc'd
==14597==    at 0x4C2A45D: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==14597==    by 0x4EA403D: getdelim (iogetdelim.c:66)
==14597==    by 0xBB73575: _nss_files_setnetgrent (stdio.h:117)
==14597==    by 0x4F4954E: __internal_setnetgrent_reuse (getnetgrent_r.c:139)
==14597==    by 0x4F49879: setnetgrent (getnetgrent_r.c:181)
==14597==    by 0x4033EA: netgroup_keys (getent.c:493)
==14597==    by 0x402370: main (getent.c:1011)

The code was very obviously buggy too:

         while (line[curlen - 1] == '\n' && line[curlen - 2] == '\\')
             /* Yes, we have a continuation line.  */
             if (found)

So if line has just the newline character, curlen will be 1 and hence, line[curlen - 2] will be the byte before the array. This kind of thing is never caught normally because of two factors:

  1. The byte before line is part of the malloc metadata
  2. The byte is read and not written to, hence there's no question of corrupting data and triggering one of malloc's consistency checking mechanisms.

Hence the code never crashes and we think that everything is just fine until the byte preceding line just happens to be 0x5b, i.e. ‘\‘. That will again never happen in the normal case since it would mean that the size of the line is at least 0x5b00000000000000 on a 64-bit system. However, even if this happens, it may not do a lot of harm since all it does is resulting in concatenating an empty string to the subsequent netgroup line.

However, consider a scenario where a netgroups file was generated in a kickstart like this:

{ for i in foo bar baz; do
    echo "${i}_netgroup \\"
    for j in $(seq 1 10); do
        echo "($i, user_$j,) \\"
    # Guess what! I can get away with the \ in the end if I leave a blank line!
    # How Cool is that!
    echo ""
done } > /etc/netgroup

Here, the administrator has found a ‘feature’ where leaving blank lines allows them to get away with their very simple script to generate netgroups with trailing backslashes. Now what happens if line happens to have a preceding 0x5b? The groups separated by the blank lines will get concatenated and all of a sudden you have one group with all of the entries being members of the first group!

So if an attacker manages to control this preceding byte in a program (possibly through a different heap overflow bug), (s)he could manipulate netgroup memberships and cause a breach. So we have a security vulnerability on our hands! Or do we?

Not such a big deal after all

So while it is fun to imagine scenarios to exploit code (who knows when you’ll get a hit!), this one seems like just a harmless buglet. Here’s why:

Controlling the preceding byte in such a manner is extremely difficult, since the malloc consistency checker should kick in before line is even allocated this offending chunk of memory. One may argue that malloc could be overridden and the preceding byte could be tailored to ones needs, but that immediately takes out the interesting attack scenarios, i.e. the setuid-root programs. If you’re doing authentication using a non-setuid command line program then you’ve got more serious issues anyway because one could simply override getnetgrent_r or similar user/group/netgroup browsing functions and bypass your check.

Finally, this ‘exploit’ depends on the netgroups file being wrong, which is again extremely unlikely and even if it is, that’s a problem with the configuration and not a vulnerability in the code.

In the end it is just a trivial off-by-one bug which must be fixed, so it was.


FUDCon Notes on my Security Exploits Session

Posted: Nov 10, 2011, 10:17

I was asked¬† by a couple of people for notes on the security exploits session that I conducted at FUDCon. I had posted the code samples on the talk page, but that is probably a little terse, so here’s a little write-up to support the code samples. To repeat what I had said multiple times earlier; I am not a security researcher, not even a security freak. This topic was suggested to me by Amit Shah, and I developed an interest in it due to my original interest, which is operating systems tools. The preparation of this talk got me interested in security, but only through the perspective of operating systems tools and programs, so I am still relatively indifferent to the subject of web-based security.

I started preparing for the session fairly late; i.e. 2 days before FUDCon. I am a little familiar with glibc code and with the way the compiler, linker, loader, etc. work on Linux, so that helped me understand a lot of the concepts behind exploits fairly easily. But concepts != working code and getting exploit code to work was the real challenge, especially when I had just about 3 evenings+nights for it. I had started with an idea of showing stack smashing and privilege escalation examples, but given the time constraint, audience level (college students) and also the constraint of my knowledge, I decided to restrict it to stack based attacks. All of the examples have a buffer which is being written to without checking for bounds of that buffer, typically with an strcpy.

The shellcode sample:

The shellcode sample as well as the final vulnerability demo (smash.c and vulnerable.c) were derived from the article Stack Smashing for Fun and Profit. That is a great article that explains in much more detail than I went into in my session, as to how the shellcode exploit can be developed.The core idea of this is:

The exploit is fairly straightforward, except that the instructions no longer work as is on Linux. These instructions require that the process image is set up in a manner that the page mapped to implement the program stack should have execute permissions. By default on recent Linux distributions (I tried this on F-15, but I am certain this should be true for at least F-13, if not earlier), the linker writes out binaries in a manner that the stack, when set up for a process, only has read and write permissions.

I spent a lot of time trying to figure out where this was set and finally found the -z option of the linker. So to write out a binary that sets up an executable stack, I had to call the linker with -z execstack. This finally enabled me to get the shellcode working.

The actual exploit

Once the shellcode was done, I could get the final vulnerability working and I immediately set about trying it. The exploit is based on the above shellcode example, except for one difference. The shellcode example is just that, an example. It is not an actual exploit; it is just a roundabout way to get a shell. The exploit I was about to do was a real crack. The idea now is to accept a string as input, which is then fed in to make a regular and buggy program provide you with a shell.

To imagine how this would work, think of the program that gives you a login prompt. In the context of this exploit, you should be able to input a crafted string into this login prompt and have it give you a shell without actually knowing the password! This is what the actual exploit ought to look like.

Again, writing the exploit was the easy part; getting it to run was quite another thing altogether. The exploit works as follows:

In all of this, there is one assumption that caused the program sample to not work; the assumption that memory maps are at predictable addresses. Recent kernels (quite some time ago actually) have a new security mechanism called Address Space Randomization which ensures that memory pages are loaded at random offsets. This meant that our educated guess would no longer work. So to be able to actually do this demo, I would have to disable address space randomization. I do that with:

echo 0 > /proc/sys/kernel/randomize_va_space

Even with this, my example would not execute by itself and would end with a SIGILL. I suspect this has something to do with the fact that my systm is x86_64 while the samples are all 32-bit. Our overflow string does not seem to agree with the instruction set on my system. In any case, it seems to run just fine inside a debugger. So if you run smash to get a shell, run gdb vulnerable and then run it with $EGG, you get the shell! At least I had a demo now.

Jump to libc

While I was trying out the shellcode example, I continued thinking about various other ways in which I could get a shell. One of the methods I thought of was to overwrite the return address with the address of the system() glibc call and pass the string via stack. I later found out via Huzaifa that this is in fact a documented way to exploit unchecked buffers on stack. Huzaifa also said that I may be missing out on something there and gave me some tips on finding the right resources for this. I still could not get this working, but at least I found out why the exploit did not work.

This exploit seemed attractive to me because it does not require an executable stack. The instructions I want to execute are already there in memory. So I only have to overwrite the return address and continue writing “/bin/sh” on the stack. I first tried with x86_64 in this case, because I was going by my own idea at that time. I soon figured out that the system() function on x86_64 did not take function arguments from stack. It took the argument from the %rdi register. My devious plan had been foiled! I did not give up however and looked at the system() implementation on i686. This retained the old behaviour of popping arguments from the stack, so my exploit was still possible here.

Not. My code was correct, but every time I run the program, the address of system() had just 3 bytes set. So it would always look something like: 0x00aabbcc. This was bad news because this meant that I cannot continue writing the shell string into the stack (strcpy stops copying when it encounters a 0x0). This means that I can call system() (like I was able to on x86_64 too), but I cannot pass it an argument. After trying enough number of times, I concluded that this must be a security feature. This was backed up by the tip Huzaifa had shared with me to (ironically) get the exploit to work. This was perhaps the first documentation of a return to libc exploit by Solar Designer. In his explanation, Solar designer mentioned that a way to fix this would be to ensure a 0x00 in the address, which is precisely what is happening here.

This obviously does not deny the fact that such an exploit can be carried out if you want to call functions that do not have arguments. Think for example, of a function that executes a shell ;)


The last modify example was a simple little trick I wrote on the last day to demonstrate how buffer overflows work and how they can be used to alter program flow. That again is not an exploit at all. At most, it can be called… a buffer overflow ;)

I had even more fun preparing for this session than actually presenting it because it taught me a lot more than I could ever have done by just reading literature. I hope those who attended my session at FUDCon enjoyed the session too.


FUDCon Pune 2011 Day 2: Me, followed by lunch, followed by me, followed by me...

Posted: Nov 07, 2011, 03:12

The title pretty much summarizes what most of my day looked like on day 2 of FUDCon. Well, not exacly, but it comes quite close. I had three sessions lined up in a single day and I was worried that I might lose my voice by the end of it. Ankur Sinha had all of 4 talks in the single day, so I was definitely better off that him.

The day started with Harish Pillay’s keynote on the community architecture team. The turnout on day 2 was less than that on day 1, which was a little surprising. Most of them trickled later in the day, so it meant that a large number of the attendees in Harish’s talk were Red Hatters and the CoEP volunteers. We probably started a little too early for a Saturday.

Immediately following that was my session on qpid messaging. The attendance in the session was modest (about 8-10 people), but the best part was that they were very involved in the session and that made the session worthwhile. Mrugesh Karnik also joined the session mid-way and asked some really good questions that actually helped my session. We ended up doing a queue design for a fictitious stock trading system and I was able to show how the design could scale very easily with a qpid messaging broker in place. Unfortunately, most of the attendees did not have laptops, so I could not engage them in a hands-on session. In fact, that was my story of the day to a large extent. I had intended all my sessions to be hands-on, but most of it never really materialized because most of the audience did not have laptops.

After the qpid session, I spent some time chatting with Sankarshan, Mrugesh, Anurag and Nisha over lunch. After that I decided to double-check my exploit code samples because it was the one session that I had never done before and it was something that is not my area of expertise. The only aspect of the exploits that I was really comfortable with was how they worked and how I could explain that using the usual tools like gdb, objdump, etc.

I was sitting in the speakers lounge cleaning up my examples when Aditya Patawari came in and asked me about my session. That reminded me that I had to actually go into the session :D We quickly left for the classroom and found pjp finishing up his python session, which had a packed audience. Once he was done, a lot of people left, which led me to think that even this talk was going to have a modest audience. However, people trickled in as I was about to begin and by the time I did begin, the room was full.

The exploits session was probably one of the best sessions I have done so far, mainly because I personally enjoyed it. The audience also consisted of people who were interested (exploits are sexy, as someone said later) and I got a lot of questions during and after the session. The talk also seemed to give some people from the audience the impression that I am a security expert, which is flattering but incorrect.

Then came the awesome part where Pai and Yogesh Babar followed up my session with impromptu sessions, which the audience lapped up eagerly as well. Pai talked about extensibility of postgresql by making it call routines in perl (typical dinosaur stuff ;) ) and Yogesh did a talk on kdump. I learnt later that Rahul Sundaram did something similar in one of the seminar halls by asking the audience to “ask him anything about Fedora and Open Source”. Pretty cool stuff.

After Pai and Yogesh were done, it was again time for me to get on to the platform for another session, this time on autotools. This was something I had done multiple times with the same examples, so it was pretty uneventful.

Day 2 was probably awaited by a lot of people for another reason – the FUDPub! We went to Park Estique near Vimaan Nagar for dinner. There was loud music and bling bling lights and food and drink. I enjoyed the food and drink; the lights gave my a headache and the loud music was, well, too loud. In any case it was fun chatting with people and having the really good food.

Like the first day, I did not get to attend any other sessions, this time for a different reason. I’ll probably submit less sessions in the next conference so that I actually get to attend other sessions and meet and talk to more people. I did meet a lot of interesting people on day 2, so all of that hectic schedule was completely worth it.


FUDCon Pune 2011: Less than 48 hours to go!

Posted: Nov 02, 2011, 18:19


and I’ve got a lot of things to do once I’m there!

FUDCon Pune 2011 has a wonderful line-up of talks and sessions going for it this weekend. I have added three workshops of my own too:

I finally got involved in preparations for FUDCon this week to try and help wherever needed. Rahul suggested I help out with the schedule arrangement, since I had some ideas on the layout of some sessions. We slugged it out for about 4 hours yesterday and another 3 hours today at the Red Hat office, trying to make sure we schedule talks in a manner that attendees can follow tracks (virtualization, security, web apps, embedded, etc.) and at the same time, have enough flexibility to add talks and sessions without having to shuffle things around all the time. Saleem has become a pro at copy-pasting across cells in spreadsheets as a result.

The end result is pretty impressive and we’re expecting even more submissions to keep people busy. There were concerns on whether there were too many sessions running in parallel, but I don’t think that matters. The event attendance is expected to be quite large, so there will e enough audience to keep all speakers busy. Besides, I don’t know a single conference where one gets to attend each and every talk in the conference. The real fun of the conference is to meet friends, collaborators and fellow geeks and not just sitting in rooms and listening to people talk.

Oh, and we have hackfests throughout the day on Sunday. The one I am particularly psyched about is Kushal Das’ libgqpid. It was an idea we had brainstormed about earlier and he wrote a lot of the code in it. Hopefully we can get a release out on Sunday with the core qpid client features ready. For the uninitiated (which is pretty much everyone I guess, since Kushal has not published the code yet), libgqpid is a glib based C wrapper around the qpid C++ client.

Looking forward to having a great time at FUDCon!