Both the kernel and the filesystem play a role. Permissions are stored in the filesystem, so there needs to be a place to store the information in the filesystem format. Permissions are enforced and communicated to applications by the kernel, so the kernel must implement rules to determine what the information stored in the filesystem means.
“Unix file permissions” refer to a traditional permission system which involves three actions (read, write, execute) controlled via three role types (user, group, other). The job of the filesystem is to store 3×3=9 bits of information. The job of the kernel is to interpret these bits as permissions; in particular, when a process attempts an operation on a file, the kernel must determine, given the user and groups that the process is running as, the permission bits of the file, and the requested operation, whether to allow the operation. (“Unix file permissions” also usually includes setuid and setgid bits, which aren't strictly speaking permissions.)
Modern unix systems may support other forms of permissions. Most modern unix systems (Solaris, Linux, *BSD) support access control lists which allow assigning read/write/excecute permissions for more than one user and more than one group for each file. The filesystem must have room to store this extra information, and the kernel must include code to look up and use this information. Ext2, reiserfs, btrfs, zfs, and most other modern unix filesystem formats define a place to store such ACLs. Mac OS X supports a different set of ACL which include non-traditional permissions such “append” and “create subdirectory”; the HFS+ filesystem format supports them. If you mount an HFS+ volume on Linux, these ACLs won't be enforced since the Linux kernel doesn't support them.
Conversely, there are operating systems and filesystems that don't support access control. For example, FAT and variants were designed for single-user operating systems and removable media and its permissions are limited to read/read-write and hidden/visible. These are the permissions enforced by DOS. If you mount an ext2 filesystem on DOS, it won't enforce the ext2 permissions. Conversely, if you access a FAT filesystem on Linux, all files will have the same permissions.
Successive versions of Windows have added support for more permission types. The NTFS filesystem was extended to store those extra permissions. If you access a filesystem with the newer permissions on an older operating system, the OS won't know about these newer permissions and so won't enforce them. Conversely, if you access an older filesystem with a newer operating system, it won't have contain of the new permissions and it is up to the OS to provide sensible fallbacks.