This looks to me like you found a stub that's intended for someone to fill in for a specific system, given that no matter what you do to call it, it will return -1 and set errno. That said, I'm not sure why they would do all the other stuff just to return -1.
You need to be looking in a different place, I think.
Perhaps the following will help you. Assuming you are using a 32-bit X86 2.6 Linux kernel .....
0x80 is the value of the software interrupt (INT) which performs the transfer to kernel mode via system_call(). See ../arch/x84/kernel/entry_32.S for the system_call() source.
The syscall interface for open() is defined in ../include/linux/syscalls.h
asmlinkage long sys_open(const char __user *filename, int flags, int mode);
The system call number for open is defined in ../arch/include/asm/unistd_32.h